17 September 2011

Reading CD-Text with VB.Net (2)

I mentioned, in the first part, that there are two control codes that you can send to get the CD-Text bytes back. We’ve used IOCTL_CDROM_READ_TOC_EX and now it’s the turn of the slightly trickier IOCTL_SCSI_PASS_THROUGH_DIRECT. This one lets us send any message we like to the device. To learn about the messages that you can send, you need to get your hands on a copy of the standard specification that defines them all. Hardware manufacturers are supposed to conform to the standard so that all the devices understand the same commands.

The standards of interest are the ones for scsi devices. You have to pay for access to the latest standards, and it looks like access to the ftp for draft standards is now restricted. The old standards are available on the internet in various places though. I still have my copies of mmc3r10g.pdf and spc2r20.pdf. It has some information that will come in handy later about the format of the bytes we have been retrieving. You can find yet more information on this format by looking at open source projects such as cdrdao, the original patent application by Sony and various other projects. The real standard is the Red Book (IEC 60908), but that costs £200 to access.

Anyway, we have to call DeviceIoControl with this other command code. We send it a structure that will contain a cdb (command data block) along with a buffer to contain the bytes that are coming back, and another buffer for so called sense-information. Sense information is helpful as it contains error information – and there are many things that can go wrong – tray out, no cd, DVD inserted, drive not ready, etc. This is probably the only reason to use this message over the other one – you can get back the sense information. It’s also useful to know how to send any command as there are a lot of them in the specification, and they don’t all have an equivalent “friendly” IOCTL code.

The command we are sending is the READ TOC command, the cdb is just 10 bytes in size. It contains one byte for the command code (&H43). Then a byte we can ignore. Then there’s a byte that holds the format. This is the same thing as the format we send with IOCTL_READ_TOC_EX – we can specify different values to get different TOC information. Again, we want to set this to 5 to retrieve CD-Text (see the spec). After that there are reserved, and inapplicable bytes (given the format). At the end there are two bytes to say how big our buffer is, and finally a control byte.

The cdb and a pointer to the databuffer go into a SCSI_PASS_THROUGH_DIRECT structure. This structure is then placed inside another structure that also holds the sense buffer. SCSI_PASS_THROUGH_DIRECT looks like this:

<StructLayout(LayoutKind.Sequential)>
Friend Structure SCSI_PASS_THROUGH_DIRECT                           ' x86   x64
    Public Length As Short                                          '  0      0
    Public ScsiStatus As Byte                                       '  2      2
    Public PathId As Byte                                           '  3      3
    Public TargetId As Byte                                         '  4      4
    Public Lun As Byte                                              '  5      5
    Public CdbLength As Byte                                        '  6      6
    Public SenseInfoLength As Byte                                  '  7      7
    Public DataIn As CommandDirection                               '  8      8
    Public DataTransferLength As Integer                            ' 12     12 
    Public TimeOutValue As Integer                                  ' 16     16
    Public DataBuffer As IntPtr                                     ' 20     24
    Public SenseInfoOffset As Integer                               ' 24     32
    <MarshalAs(UnmanagedType.ByValArray, SizeConst:=16)>
    Public Cdb() As Byte                                            ' 28     36
    '                                                        Size     44     56
    Public Shared Function GetSize() As Integer
        Return Marshal.SizeOf(GetType(SCSI_PASS_THROUGH_DIRECT))
    End Function
 
End Structure

I’ve got the offsets/sizes that I calculated in C in the comments. You can see that the IntPtr field is larger on x64. Here the cdb array is marshalled as a ByValArray which tells the marshaller to stick the bytes directly into the memory of the structure. The alternative is for the bytes to live elsewhere and for the field in the structure to be a pointer to them. We have to fill in this structure, which means we have to know what the PathId, TargetId and Lun of the CD drive are. We can get those with IOCTL_SCSI_GET_ADDRESS. The SenseInfoLength is 32. The SenseInfoOffset can be calculated at runtime using Marshal.OffsetOf to get the offset of the sense buffer in the outer structure (see attached project) – it will vary on x86 and x64.

As before, we’ll call once with a data buffer of size 2. On return, our data buffer will contain the size of the data (excluding the field that holds the size itself). So we can then allocate precisely as much memory as is required to hold the whole CD-Text.

Option Strict On
Option Explicit On
 
Imports CdTextIoCtl.NativeMethods
Imports Microsoft.Win32.SafeHandles
Imports System.ComponentModel
Imports System.IO
Imports System.Runtime.InteropServices
Imports System.Text
 
Public Class CdTextRetriever
 
    Public Shared Function GetCdText(driveInfo As DriveInfo) As Object
        Dim cdText As Object = Nothing ' don't know what the return will look like yet.
        Dim devicePath As String = String.Format("\\.\{0}:", driveInfo.Name(0))
        Const FILE_ATTRIBUTE_NORMAL As UInteger = &H80
        Const GENERIC_READ As UInteger = &H80000000UI
        Const GENERIC_WRITE As UInteger = &H40000000
        Const FILE_SHARE_READ As UInteger = 1
        Const FILE_SHARE_WRITE As UInteger = 2
        Const OPEN_EXISTING As UInteger = 3
        Using hDevice As SafeFileHandle = NativeMethods.CreateFile(devicePath,
                GENERIC_READ Or GENERIC_WRITE,
                FILE_SHARE_READ Or FILE_SHARE_WRITE,
                IntPtr.Zero, OPEN_EXISTING,
                FILE_ATTRIBUTE_NORMAL, IntPtr.Zero)
            If hDevice.IsInvalid OrElse hDevice.IsClosed Then Throw New Win32Exception
            ReadCdText(hDevice)
        End Using
        Return cdText
    End Function
 
 
    Private Shared Sub ReadCdText(hDevice As SafeFileHandle)
        Dim address As SCSI_ADDRESS = GetScsiAddress(hDevice)
        ' like in the other program...get the length
        Dim bytes(1) As Byte
        GetCdTextBytes(hDevice, address, bytes)
        ' again we add 2 for the length parameter itself so it's length - 1 + 2 to size the array
        bytes = New Byte((CType(bytes(0), Integer) << 8) Or bytes(1) + 1) {}
        GetCdTextBytes(hDevice, address, bytes)
        Dim sb As New StringBuilder
        For i As Integer = 4 To bytes.Length - 19 Step 18
            For j As Integer = 0 To 17
                sb.Append(bytes(i + j).ToString("X2"))
                If i = 3 OrElse i = 16 Then sb.Append("  ")
            Next
            sb.AppendLine()
        Next
        My.Computer.FileSystem.WriteAllText(IO.Path.Combine(My.Computer.FileSystem.SpecialDirectories.MyDocuments, IO.Path.GetRandomFileName), sb.ToString, False)
    End Sub
 
    Private Shared Function GetScsiAddress(hDevice As SafeFileHandle) As SCSI_ADDRESS
        Dim address As New SCSI_ADDRESS
        Dim bytesReturned As Integer
        Const IOCTL_SCSI_GET_ADDRESS As Integer = &H41018
        Dim result As Boolean = NativeMethods.DeviceIoControl(hDevice,
                IOCTL_SCSI_GET_ADDRESS, IntPtr.Zero, 0, address,
                SCSI_ADDRESS.GetSize, bytesReturned, IntPtr.Zero)
        If result = False Then Throw New Win32Exception
    End Function
 
    Private Shared Sub GetCdTextBytes(hDevice As SafeFileHandle, address As SCSI_ADDRESS, bytes() As Byte)
        Dim bytesReturned As Integer
        Dim sptdwb As New SCSI_PASS_THROUGH_DIRECT_WITH_BUFFER
        Dim pinnedBytesHandle As GCHandle = GCHandle.Alloc(bytes, GCHandleType.Pinned)
        sptdwb.ScsiPassThroughDirect.cdb = New Byte(15) {} ' The cdb as described in the standard
        sptdwb.SenseBuffer = New Byte(31) {}
        sptdwb.ScsiPassThroughDirect.Length = CShort(SCSI_PASS_THROUGH_DIRECT.GetSize)
        sptdwb.ScsiPassThroughDirect.CdbLength = 10
        sptdwb.ScsiPassThroughDirect.DataIn = CommandDirection.SCSI_IOCTL_DATA_IN
        sptdwb.ScsiPassThroughDirect.DataTransferLength = bytes.Length
        sptdwb.ScsiPassThroughDirect.DataBuffer = pinnedBytesHandle.AddrOfPinnedObject
        sptdwb.ScsiPassThroughDirect.TimeOutValue = 10
        sptdwb.ScsiPassThroughDirect.SenseInfoOffset = Marshal.OffsetOf(GetType(SCSI_PASS_THROUGH_DIRECT_WITH_BUFFER), "SenseBuffer").ToInt32
        sptdwb.ScsiPassThroughDirect.SenseInfoLength = 24
 
        sptdwb.ScsiPassThroughDirect.PathId = address.PathId
        sptdwb.ScsiPassThroughDirect.TargetId = address.TargetId
        sptdwb.ScsiPassThroughDirect.Lun = address.Lun
        ' CDB is set with the READ_TOC command
        sptdwb.ScsiPassThroughDirect.cdb(0) = &H43
        sptdwb.ScsiPassThroughDirect.cdb(2) = 5
        sptdwb.ScsiPassThroughDirect.cdb(7) = CByte((bytes.Length >> 8) And &HFF)
        sptdwb.ScsiPassThroughDirect.cdb(8) = CByte(bytes.Length And &HFF)
        Try
            Const IOCTL_SCSI_PASS_THROUGH_DIRECT As Integer = &H4D014
            Dim result As Boolean = NativeMethods.DeviceIoControl(hDevice,
               IOCTL_SCSI_PASS_THROUGH_DIRECT,
               sptdwb,
               Marshal.SizeOf(sptdwb),
               sptdwb,
               Marshal.SizeOf(sptdwb),
               bytesReturned,
               IntPtr.Zero)
            If result = False Then Throw New Win32Exception
        Finally
            pinnedBytesHandle.Free()
        End Try
    End Sub
 
End Class

So, that’s another way to read the same bytes. I’ve told this one to save a file with a dump of the bytes into MyDocuments. It splits the data into lines with 18 bytes on each. It starts at byte 4 to skip the length and two other bytes which appear before the Pack Data. Pack Data contains 12 bytes of information each with a header describing the information and a 2 byte CRC code at the end. Most of the information will be ascii text, although some packs don’t contain text. The header contains a pack type code which has things like Title, Songwriter, etc. It also has values to identify the track number that the data is for. It’s quite a simple format, but tricky to decode without a precise definition of the format! These projects are a bit naff – I’ve not used VS2010 much and it’s messing up my solution with 2 projects in. I can’t seem to get it so that you can switch easily between x86 and AnyCpu builds, so it ends up in mixed mode. You can just start a new empty solution and add the relevant bits if it wont build and run.

Next time I’ll decode the bytes, check the crcs and the sense information. I’ve found that occasionally I get some corrupt data coming in the first time it reads – hopefully the CRC will fail and I can retry.

11 September 2011

Reading CD-Text with VB.Net (1)

Note: most commercial CDs do not have any CD Text on them.

There are probably a few ways to read the CD Text. I’m going to look at using the windows api function DeviceIOControl function. The steps are (1) Get a device handle (2) Prepare a message (3) Send the message to the device (4) Decode the results.

To get a device handle, you call CreateFile sending the drive letter of the drive formatted as a device path. You also have to specify the minimum permissions you will be needing for the operations you will be performing. If you fail to ask for a handle with sufficient permissions then you will get errors later on when you are calling DeviceIOControl.

In .Net you can pick various types to represent handles. I’ve seen people use Integer – which isn’t good as Integer is always 32 bits whereas handles may be 64 bits. IntPtr is a better choice as it will match the size of the C HANDLE type it is representing. HandleRef is another good choice, this is used to prevent the garbage collector from destroying your handle before a call ends – which would be embarrassing. SafeHandle is the best choice to use as it ensures that the handle will be closed in certain unusual circumstances. SafeHandle is an abstract class, there are various concrete implementations. For a device handle we can use the SafeFileHandle.

The IOCTL_CDROM_READ_TOC_EX control code can be used to get the CD-Text back from the drive. Another alternative is to use a less specific control code such as  IOCTL_SCSI_PASS_THROUGH which lets you send any command to the drive, it’s a bit more involved – I’ll show how to use that later. When you are working from scratch, and trying to figure this stuff out for yourself, it helps to have a C++ project that you can use to dump out the values of the constants, size of structures, and offsets of structure fields. A bit of work in C++ can save a lot of time searching the internet with vague error messages like “A call to PInvoke function xyz has unbalanced the stack”.

If you want to set up a C++ project then the steps are:
1) Download and install the WDK. It’s a big download. The bit you need are the “Build Environments”, as selecting this will ensure that the header files are installed.
2) Start visual studio 2010 and a new C++ CLR Console application project called, say, StructSizes
3) Right click the project name in the solution explorer, click References, Add New Reference and add System.Windows.Forms.
4) Right click the project name in the solution explorer, click Properties, Drill down to Configuration Properties/VC++ Directories and then select Include Directories. Edit the string to include the path to the wdk header files. This will involve adding a “;” and then the path. Mine looks like this after the edit: $(VCInstallDir)include;$(VCInstallDir)atlmfc\include;$(WindowsSdkDir)include;$(FrameworkSDKDir)\include;C:\WinDDK\7600.16385.1\inc\api
5) As an example lets dump out the value of the IOCTL command and some info about a structure. This goes into the cpp file with the same name as the project, so StructSizes.cpp.

#include "stdafx.h"
#include <Windows.h>
#include <winioctl.h>
#include <ntddcdrm.h>
#include <ntddscsi.h>
#include <stddef.h> 
 
using namespace System;
using namespace System::Text;
using namespace System::Windows::Forms;
 
[STAThread()] // to make the clipboard happy.
int main(array<System::String ^> ^args)
{
    StringBuilder ^sb = gcnew System::Text::StringBuilder();
    sb->AppendLine(L"IOCTL_CDROM_READ_TOC_EX: " + (IOCTL_CDROM_READ_TOC_EX).ToString(L"x2"));
    sb->AppendLine(L"SCSI_PASS_THROUGH offsets + size:");    
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::Length)).ToString() + L" Length");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::ScsiStatus)).ToString() + L" ScsiStatus");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::PathId)).ToString() + L" PathId");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::TargetId)).ToString() + L" TargetId");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::Lun)).ToString() + L" Lun");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::CdbLength)).ToString() + L" CdbLength");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::SenseInfoLength)).ToString() + L" SenseInfoLength");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::DataIn)).ToString() + L" DataIn");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::DataTransferLength)).ToString() + L" DataTransferLength");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::TimeOutValue)).ToString() + L" TimeOutValue");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::DataBufferOffset)).ToString() + L" DataBufferOffset");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::SenseInfoOffset)).ToString() + L" SenseInfoOffset");
    sb->AppendLine((offsetof(SCSI_PASS_THROUGH, SCSI_PASS_THROUGH::Cdb[0])).ToString() + L" Cdb[0]");
    sb->AppendLine(L"**TOTAL SIZE: " + (sizeof(SCSI_PASS_THROUGH)).ToString());
    sb->AppendLine();
    sb->AppendLine();
    Console::WriteLine(sb->ToString());
    Clipboard::SetText(sb->ToString());
    Console::ReadKey();
    return 0;
}

6) It should build and run. It sets the clipboard with the text. You can’t dump out the offset of bit fields, like you get in the CDROM_READ_TOC_EX structure. You should dump out x86 and x64 versions so that you can check your .Net structures match on both.

Let’s say you’ve made your .Net structure, called ScsiPassThrough, and want to see the offset of the Length field and the size. You would use the SizeOf and OffsetOf methods of the Marshal class which is in the System.Runtime.InteropServices namespace. e.g.:

Console.WriteLine(Marshal.OffsetOf(GetType(ScsiPassThrough), "Length").ToString)
Console.WriteLine(Marshal.SizeOf(GetType(ScsiPassThrough)))

Ok, back to the task in hand. We want to send IOCTL_CDROM_READ_TOC_EX which has the value &H24054. The documentation says we have to send a structure CDROM_READ_TOC_EX to specify what exactly we want to get back from the TOC.

typedef struct _CDROM_READ_TOC_EX {
    UCHAR Format    : 4;
    UCHAR Reserved1 : 3; // future expansion
    UCHAR Msf       : 1;
    UCHAR SessionTrack;
    UCHAR Reserved2;     // future expansion
    UCHAR Reserved3;     // future expansion
} CDROM_READ_TOC_EX, *PCDROM_READ_TOC_EX;

This structure has a size of 4 bytes on x86 and x64. We need to set the Format field to CDROM_READ_TOC_EX_FORMAT_CDTEXT (value 5) to say we want to retrieve CD Text. All the others will be 0. The structure is made up of 4 bytes. SessionTrack, Reserved2 and Reserved3 get a byte each. The other byte is divided up into 3 fields – Format, Reserved1 and Msf. Format takes the lo 4 bits of the byte. Reserved takes the next 3 and Msf takes the hi bit. We can’t split fields up in .Net like that. We can either declare a structure that has 4 byte fields, and expose the three bit fields with properties, or, as we just want to set the one field, we could also just send an integer with the correct bit pattern - an integer has the same size and all the dll that we are calling cares about is that it gets 4 bytes with the correct bits set. I came up with this:

<StructLayout(LayoutKind.Sequential)>
Friend Structure CDROM_READ_TOC_EX
    Public FormatReservedMsf As Byte
    Public SessionTrack As Byte
    Public Reserved2 As Byte
    Public Reserved3 As Byte
 
    ' properties will just read the public fields without contributing to the size of the structure.
    Public ReadOnly Property Format As ReadTocExFormat
        Get
            Return CType(FormatReservedMsf And &HF, ReadTocExFormat)
        End Get
    End Property
 
    Public ReadOnly Property Reserved1 As Integer
        Get
            Return (FormatReservedMsf >> 4) And &H7
        End Get
    End Property
 
    Public ReadOnly Property Msf As Boolean
        Get
            Return (FormatReservedMsf And &H80) = &H80
        End Get
    End Property
 
    Public Sub SetFormatReservedMsf(format As ReadTocExFormat, msf As Boolean)
        FormatReservedMsf = CByte(format Or If(msf, &H80, 0))
    End Sub
 
    Public ReadOnly Property Size As Integer
        Get
            Return 4
        End Get
    End Property
 
End Structure

The documentation says that when we send the message we should also send a buffer. On return the buffer will contain a CDROM_TOC_CD_TEXT_DATA structure, followed by an array of CDROM_TOC_CD_TEXT_DATA_BLOCK structures. So – how big should the buffer be to hold all this? The answer is that we don’t know. We can either create a huge buffer that will be big enough, or we can call once to read the CDROM_TOC_CD_TEXT_DATA.Length field and use that to size the buffer. The buffer can be an array of bytes – they are easy to marshal if one of the parameters states the size of the array. DeviceIOControl will look like this:

<DllImport("kernel32", SetLastError:=True)>
Public Shared Function DeviceIoControl(hVol As SafeFileHandle, controlCode As Integer,
   ByRef inbuffer As CDROM_READ_TOC_EX, inBufferSize As Integer,
   <MarshalAs(UnmanagedType.LPArray, SizeParamIndex:=6)> ByVal outBuffer() As Byte,
   outBufferSize As Integer,
   <Out()> ByRef bytesReturned As Integer, ovelapped As IntPtr) As Boolean
End Function

Note the structure goes ByRef – it’s a value type and we don’t want to send the whole structure to the function as it is expecting a pointer to the structure. Passing it ByRef tells the marshaller to send a pointer. The array, being a reference type, goes ByValue.

So the first time we call it we want to just get the Length field which is a WORD field, 2 bytes. Then we can call it again with a byte array that is this Length field + 2 as the value of the Length field is the size not including itself. The bytes are the wrong way round to read as a UShort with the BitConverter. I guess you could reverse the array. At this point my program gets the bytes and doesn’t do anything with them…

Imports System.IO
Imports System.Runtime.InteropServices
 
Public Class CdTextRetriever
 
    Private Const IOCTL_CDROM_READ_TOC_EX As UInteger = &H24054
 
    Public Shared Function GetCdText(driveInfo As DriveInfo) As Object
        Dim cdText As Object = Nothing ' don't know what the return will look like yet.
        Dim devicePath As String = String.Format("\\.\{0}:", driveInfo.Name(0))
        Const FILE_ATTRIBUTE_NORMAL As UInteger = &H80
        Const GENERIC_READ As UInteger = &H80000000UI
        Const GENERIC_WRITE As UInteger = &H40000000
        Const FILE_SHARE_READ As UInteger = 1
        Const FILE_SHARE_WRITE As UInteger = 2
        Const OPEN_EXISTING As UInteger = 3
        Using hDevice As SafeFileHandle = NativeMethods.CreateFile(devicePath,
                GENERIC_READ Or GENERIC_WRITE,
                FILE_SHARE_READ Or FILE_SHARE_WRITE,
                IntPtr.Zero, OPEN_EXISTING,
                FILE_ATTRIBUTE_NORMAL, IntPtr.Zero)
            If hDevice.IsInvalid OrElse hDevice.IsClosed Then Throw New Win32Exception
            ReadCdText(hDevice)
        End Using
        Return cdText
    End Function
 
 
    Private Shared Sub ReadCdText(hDevice As SafeFileHandle)
        Dim size As Integer = GetCdTextSize(hDevice)
        Dim bytes(size - 1) As Byte
        ReadCdText(hDevice, bytes)
        Debugger.Break()
        ' TODO: Decode the bytes!
    End Sub
 
    Private Shared Function GetCdTextSize(hDevice As SafeFileHandle) As Integer
        Dim bytes(1) As Byte ' 2 bytes to read the size of the buffer required
        ReadCdText(hDevice, bytes)
        Return (CType(bytes(0), Integer) << 8) Or bytes(1) + 2
    End Function
 
    Private Shared Sub ReadCdText(hDevice As SafeFileHandle, bytes() As Byte)
        Dim request As New CDROM_READ_TOC_EX
        request.SetFormatReservedMsf(ReadTocExFormat.CDROM_READ_TOC_EX_FORMAT_CDTEXT, False)
        Dim returned As Integer = 0
        Dim result As Boolean = DeviceIoControl(hDevice, IOCTL_CDROM_READ_TOC_EX, request,
                                                 request.Size, bytes, bytes.Length,
                                                 returned, IntPtr.Zero)
        If result = False Then Throw New Win32Exception
    End Sub
 
End Class