Ultimately there is no scientific debate on whether or not the mean average of global climate temperature is rising.

Given this information one immediately jumps to the conclusion that this increase is drastic, widespread and spells out the end of the world or some such nonsense.  Luckily the end of the world isn’t scheduled for Dec. 21 2012, and not only are temperature increases non-uniform, there are places on the globe which have been getting cooler since we have began to measure more accurately.

Whereas many parts of the world are experiencing an overall increase in temperature, some parts of the southeastern U.S and parts of the North Atlantic have decreased in temperature over the last 100 years.

What is also interesting is that while the Arctic ice is decreasing, the Antarctic ice sheet is actually growing slightly.  A quote from the NCDC:

Sea ice in the Antarctic has shown very little trend over the same period, or even a slight increase since 1979.

Another misconception that countless individuals have is this notion that melting glaciers are somehow responsible for the global mean sea level rising.  The rise we’ve seen over the last several thousand years is due in large part to thermal expansion.  Even today, modern science still does not fully understand how glacier dynamics play into the hydrologic system.  A quote from the NCDC:

… our current understanding of ice sheet dynamics renders uncertainties too large to be able to assess the likelihood of large-scale melting of these ice sheets.

The moral of this story is that while global warming is happening there are simply too many unknowns to make sweeping statements about glacier melt, rising sea levels and the overall impact on the various species which inhabit our planet.

So whats the story with Dell vs. HP?  Each company has created their own respective bootable, CD/DVD based firmware update utilities.  HP with Firmware Maintenance CD and Dell with Dell Systems Build and Update Utility, Proliant and PowerEdge class systems respectively.

When did these manufacturers first come out with this bootable media?

HP, December 2003.

Dell, November 2007.

Why did Dell take so long?  They didn’t.

Here’s my take.  HP has been in the mainframe and unix world for a very long time.  They have been making carrier-grade hardware for as long as I can remember.  Dell is “relatively” the new kid on the block with the server-scene.  While they do have some catching up to do, they’re making good progress with the new PE 1950 III’s.  It is nice to see servers finally using 2.5″ drives that run 10KRPM.  Packing 6 drives into a 1U server without compromising speed or function is pretty impressive.

Ultimately the moral of this story is, as of about a year ago Dell is finally making bootable media that assists admins in updating firmware whereas HP has been doing it since 2003.  Let’s hope Dell keeps up the pace as I really like the stability of their hardware.  RHEL5 and CentOS5 run like a dream on PowerEdge hardware.

Heres the config with comments:

[client]
port = 3306
socket = /var/lib/mysql/mysql.sock

[mysqld]
port = 3306
datadir = /var/lib/mysql/
socket = /var/lib/mysql/mysql.sock
skip-locking
skip-name-resolve
key_buffer = 256M
max_allowed_packet = 2M
table_cache = 512
sort_buffer_size = 64M
read_buffer_size = 64M
read_rnd_buffer_size = 32M
myisam_sort_buffer_size = 32M
thread_cache_size = 32
query_cache_size = 512M
# Dual procs, dual core, 4*2=8
thread_concurrency = 8
max_connections = 1000

# dont make mistakes with engine
default-storage-engine = InnoDB

log-slow-queries = /var/log/mysql-slow.log
long_query_time = 5

# Replication Master Server (default)
# binary logging is required for replication
log-bin=mysql-bin
relay-log=relay-bin

log_slave_updates = OFF

# we’re the master
server-id = 1

# heres one you should use: innodb_file_per_table
# ultimately this makes innodb create a file on disk for each table you create.
# this is incredibly useful if you have to re-claim disk space. if you don’t use this
# and you drop a table, the ibdata1 doesn’t shrink. you lose the diskspace.
innodb_file_per_table
innodb_data_home_dir = /var/lib/mysql/
innodb_data_file_path = ibdata1:10M:autoextend
# now i could put the logs on a different partition/disk, but it hasn’t been a bottleneck so far
innodb_log_group_home_dir = /var/lib/mysql/
innodb_log_arch_dir = /var/lib/mysql/
innodb_log_files_in_group = 2
# You can set .._buffer_pool_size up to 50 - 80 %
# we have more than 8G on the box.
innodb_buffer_pool_size = 4096M
innodb_additional_mem_pool_size = 512M
# Set .._log_file_size to 25 % of buffer pool size
innodb_log_file_size = 1024M
innodb_log_buffer_size = 128M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[myisamchk]
key_buffer = 128M
sort_buffer_size = 128M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
interactive-timeout

Now this setup isn’t for everyone, and I left out the replication bits. You will undoubtedly have to tweak all the memory utilization settings for your own config, but this one works for me. Really the moral of this story is, use the following if you can help it:

• CentOS 64bit
• RAID (preferably 10 for speed/performance)
• Partition strictly for MySQL using XFS as the filesystem
• InnoDB with table per/file

Enjoy.

hexlate

The first one I call “hexlate”.  hexlate is just hexdump with special formatting arguments.

#!/bin/sh
hexdump -v -e '"\\""x" 1/1 "%02x"'

Essentially here we run hexdump(1), pass it “-v” which causes hexdump to display all input data and not replace duplicate data with asterisks.  Next is the formatting rule.  Here I tell it that I want to display all characters in hexadecimal code in two character format.

Here is a test file:

asgar:tmp evan$ cat old
ABCDE
asgar:tmp evan$

Notice the carriage at the end.

Here is the same test file run through hexlate:

asgar:tmp evan$ cat old | ./hexlate
\x41\x42\x43\x44\x45\x0aasgar:tmp evan$

Notice the \x0a at the end? Thats the carriage return.

asgar:tmp evan$ echo -ne '\n' | ./hexlate
\x0aasgar:tmp evan$

hexlate is a really useful tool to quickly eyeball binary data, and then use it with perl.  In this format, we can use the print function in perl to produce the binary output via stdout.

perl -e print “\x41\x42\x43\x44\x45\x0a”

asgar:tmp evan$ perl -e 'print "\x41\x42\x43\x44\x45\x0a"'
ABCDE
asgar:tmp evan$

If you had a large block of hexlate output, you could easily incorporate it into a script.

prep, compare

The next two tools are primarily used together, “prep” and “compare”.

The first one, prep, is (once again) another hexdump command line:

#!/bin/sh
hexdump -v -e '1/1 "%02x\n"' $1

This time the format looks like this:

asgar:tmp evan$ cat old | ./prep
41
42
43
44
45
0a
asgar:tmp evan$

Why would this be useful at all?  I’ll tell you why, lets say you have two binary strings that you want to compare, literally.  If you simply did a hexdump against them, or even a hexlate the output can be a bit difficult to easily spot changes.  Let’s take a look at “compare” and see where I am going with this.

Here is what “compare” looks like:

export PATH=./:${PATH}
diff -yW10 <(prep $1) <(prep $2)

Here’s how to use it and the output:

asgar:tmp evan$ ./compare ./old ./new
41      41
42      42
43      43
44  |   20
45      45
0a      0a

Whoa wtf?  Yeah, rad huh?  Okay so here is whats happening.  Lets start with the <(prep $1) and <(prep $2) action.  In bash we can take the output of a command and turn it into a file, /dev/fd/<number>.  When we do this, we get output as files.  So we “prep” file “old” and we “prep” file “new”.  While diff(1) can only will compare files, we’ve just turned the stdout into files so there is no problem.  Next, the diff is passed the arguments “-y”, this forces the diff to be done as a “side-by-side” in two columns.  That is how we get the files to line up.  The other argument is “-W10″, this just sets the width to 10 possible print columns.

Ultimately we produce output side-by-side, and it is super easy to spot the difference.

File “old” is “ABCDE\n” and file “new” is “ABC E\n”.  The letter “D” was replaced with a literal space ” “.  We can see the change clear as day.

I hope you find these tools as useful as I have.  Granted they’re pretty simple command lines, but damn are they useful.

Disclaimer: If you end up destroying a magstripe on a card you really care about, don’t come running to me.  erase.pl is to be used with caution.  I will not be held responsible for any fuck-ups you might make with these tools.

One last note before we get started.  I will not answer any questions regarding Device::SerialPort, debugging or any questions regarding serial ports.  If you can’t figure out how to get the MSR206 to talk to your serial port, I sure as hell am not going to hand-hold you through a duplication process.

Now that you’ve been adequatedly taunted and warned, lets have some fun.

Here is the README from the archive:

#
# dmsr-2.1 README
## It is important to know that only ISO card format is supported
# at this time.  We use the 'read' functionality as opposed to the
# 'read raw' functionality.
#
# Evan Brewer <dmess0r@gmail.com> - 2008/09/22
1. Install Device::SerialPort (CPAN works)
perl -MCPAN -e 'install Device::SerialPort'


2. Copy card data
./read.pl --device=<device> > card.read
[Wait for orange]
[Swipe]


3. Convert and write out data to MSR206
cat card.read | ./dumpdata.pl | ./strip.pl | ./write.pl --device=<device>
[Wait for orange]
[Swipe]


4. Compare
./read.pl --device=<device> > blank
[Wait for orange]
[Swipe]
diff costco.read blank

There is a typo on the third step in the README.  I ended up testing a costco card, and while I was writing my notes I left it in.  Oops.  Should read something like:

diff card.read blank

Whatever, you get the point.

Quite terse, here is the process:

1. Install Device::SerialPort perl module.  Mandatory.
2. Hook up your MSR206.
3. Read the card data out to a file via stdout: ./read –device=/dev/ttyUSB0 > card.read
4. Wait for middle orange light
5. Swipe
6. If green light after swipe, then your read is good, check file card.read for data
7. Erase a blank to which you’ll copy the new data.
8. ./erase.pl –device=/dev/ttyUSB0
9. Wait for orange light
10. Swipe
11. Run dumpdata.pl against to pull out all three track data: cat card.read | ./dumpdata.pl > card.data
12. Take the output of card.data, and run it through strip.pl to remove the sentinels: cat card.data | ./strip.pl > card.strip
13. Finally, push the dumped, stripped card data into write.pl on the blank card: cat card.strip | ./write.pl –device=/dev/ttyUSB0
14. Wait for orange light
15. Swipe
16. If green light after swipe, then your write is good (probably).
17. Now, to make sure, you’ll want to perform a re-read on the blank and then compare the output against the original read to make sure your dupe is good.
18. ./read –device=/dev/ttyUSB0 > card.re-read
19. Wait for orange light
20. Swipe
21. diff card.read card.re-read
22. Presto, you’ve just duplicated an ISO magstripe.

Cool kids will just string the entire write process together.

cat card.read | ./dumpdata.pl | ./strip.pl | ./write.pl --device=/dev/ttyUSB0

Your mileage may vary, but the tools work for me.

Also, if your MSR206 ends up with a green/red combo illuminated, it means that whatever action was performed didn’t go so hot.  Usually happens on a read of a card that contains data which is not in the expected ISO format.  The fast way to reset your MSR206 is to use ./reset –device=/dev/ttyUSB0.  This will take a second or three and it will reset your device back to a state where you can perform reads/writes again.

One of these days I’ll clean up dmsr206.pl to the point where it is an actual reference script that contains all of the functionality that is available in the reference manual.

Enjoy!

Traffic Message Channel, or TMC, is a method, or technology by which traffic data is delivered to receivers which then convert this data to pertinent, geographic-based traffic-information.  While there are a handful transmission methods on which TMC is delivered such as XM and MSN Direct, the only one I will cover today is the FM based method: Radio Broadcast Data System.

Radio Broadcast Data System (RBDS) is the official name in the USA for Radio Data System, the communications protocol from the European Broadcasting Union.  One of the only differences, minor as it is, is the numbering system assigned to the 31 formats. More detail on RDS may be found in European Standard 50067.

The third harmonic of the pilot tone of stereo FM (19 kHz), 57 kHz, was chosen as the subcarrier to carry data at 1187.5 bps, or a clock frequency of 1.11875 kHz.  There also is a maximum deviation of +/- 0.125 bps.   The third harmonic was chosen to avoid any interference with stereo or its second harmonic, the stereo difference frequency of 38 kHz.

The amplitude of the wave is decoded as binary, the resulting data is essentially square.  A very simple 1/0 implementation.

The “Block” is the smallest unit of data in an RBDS transmission with 26 bits.  The first portion of the Block structure is the “Data”, comprising of 16 bits.  The second, the “Checkword”, comprising of 10 bits.

[ Data (16bits) ][ Checkword (10 bits) ]

The Checkword is a checksum against the Data included in the Block to assist in error protection.

A “Group” comprising of 4 Blocks is 104 bits wide.

[ Block 1 ][ Block 2 ][ Block 3 ][ Block 4 ]

Block 1 contains:

[ PI Code ][ Checkword ]

The PI Code is the Program Identifical code which uniquely identifies the transmitting radio station.

Block 2 contains:

[ Group code ][ B0 ][ TP ][ PTY ][ 5 bits ][ Checkword ]

Block 3 contains:

[ Data ][ Checkword ]

Block 4 contains:

[ Data ][ Checkword ]

The group code identifies the type of message as RDS, such as 8A (RDS-TMC).  B0 is the code for the version, a ‘0′ for RDS version ‘A’, or ‘1′ for RDS version ‘B’.  TP stands for Traffic Program and identifies whether or not the station is able to send traffic announcements (TA).  We don’t care about TA.  PTY is used for describing the Program Type.

For a comprehensive view of the RDS data structure, refer to RDS Basics by 2wcom.

TMC

Traffic Message Channel is one of many information fields contained in RBDS and is described in detail in 14819-1.  It is digitally encoded, contains traffic data and is processed with the Alert-C protocol.  The data broadcast supports only very lightweight encryption for commercial services and is described in 14819_6.

RDS-TMC (TMC) data is transmitted in group 8A.  Pulled from ISO 14819-1, here is the structure of 8A:

Block 1 contains:

[ PI Code ][ Checkword ]

The PI Code is the Program Identifical code which uniquely identifies the transmitting radio station.

Block 2 contains:

[ Group code ][ B0 ][ TP ][ PTY ][ T ][ F ][ DP ][ Checkword ]

Group code through PTY is the same as listed above in the RDS specification.  T, F, and D are used together to indicate whether this is part of a Multi Group message. TFD = 001 indicates that this is the first group in a sequence, or TFD = 000 indicates that this group is NOT the first.  DP stands for ‘duration’ and ‘persistence’, essentially indicates the timeframe related to the traffic event so that the decoder knows whether or not this event is ‘fresh’.

Block 3 contains:

[ D ][ PN ][ Extent ][ Event ][ Checkword ]

D indicates whether or not diversion advice is necessary. PN stands for ‘positive’ and ‘negative’, which indicates in which direction of the queue events and is opposite to the road direction.  Extent indicates the ‘extension’ of the even and is measured by Location Table entries.  Event, the second longest code is cross-referenced against a lookup-table in the Event Code table.

Block 4 contains:

[ Location ][ Checkword ]

Location, the longest code is cross-referenced against the Location Table database.

Event Codes

If you’re interested in the Event codes, you should check out 14819-2.  There are some pretty interesting codes you’ll notice.  There are some pretty basic ones, such as, Line 37 “traffic congestion, average speed of 10 km/h”, or Line 154 “closed ahead. Stationary traffic for 4 km”.  There also are codes which are geographically specific such as Line 1128 “bull fight”, or Line 1129 “cricket match”.

Hacking

Andrea Barisani and Daniele Bianco of inversepath.com back in 2007 wrote up an article for Phrack on just this subject.  Also at the Telemobility Conference back in 2007, they presented this talk.

Practical usage for the US

So what does all of this mean for the average Joe in the US.  It is important to know that at the time of this writing, RDS-TMC is not embraced like you’d think it SHOULD be.  It is a fantastic system and it is gaining momentum but it is not available everywhere you go.  The official site for the US is realtimetraffic.net and this site contains the most current information regarding information for the US.  This Station Listing of RDS-TMC channels in the US will give you an idea of the coverage in the US thus far.

So here we go, torrent from the workstation, save on mybook, and stream onto series3?  Seems like a good idea.

First I had to hack the mybook and get SSH installed to make any of this work.  I used the SSH enabler hack found here:

http://www.mybook-linux.co.nr/

Next I needed to get Python and ffmpeg installed as they are dependencies of pyTivo.  I quickly checked the architecture and memory available.

Linux MyBookWorld 2.6.17.14 #1 PREEMPT Mon Mar 26 12:36:33 BST 2007 armv5tejl GNU/Linux

Now this processor is decent, but not blazingly fast so I decided to grab binaries rather than compile from source.  Believe it or not the mybook came preinstalled with gcc and binutils.  Mind-bending no?

So I setup Optware and installed python25 and ffmpeg via ipkg.

[root@MyBookWorld ~]# python -V
Python 2.5.2

So far so good.

[root@MyBookWorld ~]# ffmpeg -version
FFmpeg version UNKNOWN, Copyright (c) 2000-2008 Fabrice Bellard, et al.
configuration: –enable-cross-compile –cross-prefix=/home/slug/optware/gumstix1151/toolchain/gumstix-buildroot/build_arm_nofpu/staging_dir/bin/arm-linux-uclibc- –arch=arm –disable-encoder=snow –disable-decoder=snow –enable-shared –disable-static –enable-gpl –enable-postproc –prefix=/opt
libavutil version: 49.6.0
libavcodec version: 51.54.0
libavformat version: 52.13.0
libavdevice version: 52.0.0
built on Apr 15 2008 23:06:38, gcc: 3.4.5
FFmpeg UNKNOWN
libavutil   3212800
libavcodec  3356160
libavformat 3411200
libavdevice 3407872


Next I tested ffmpeg to make sure it was able to transcode properly.

[root@MyBookWorld tmp]# ffmpeg -i test.mp4 test.mp2
FFmpeg version UNKNOWN, Copyright (c) 2000-2008 Fabrice Bellard, et al.
configuration: –enable-cross-compile –cross-prefix=/home/slug/optware/gumstix1151/toolchain/gumstix-buildroot/build_arm_nofpu/staging_dir/bin/arm-linux-uclibc- –arch=arm –disable-encoder=snow –disable-decoder=snow –enable-shared –disable-static –enable-gpl –enable-postproc –prefix=/opt
libavutil version: 49.6.0
libavcodec version: 51.54.0
libavformat version: 52.13.0
libavdevice version: 52.0.0
built on Apr 15 2008 23:06:38, gcc: 3.4.5
Illegal instruction

Shit.  Illegal instruction.  This means that the binary is probably attempting to use a register which undoubtedly doesn’t exist, OR it is making a call to a processor extension which isn’t present.  I know that ffmpeg(1) makes significant usage of processor extensions such as MMX MMX2 SSE and SSE2, and this binary package was clearly built without the necessary –disable flags.  Unless you explicitly compile ffmpeg without those extensions, they will be used.

Processor       : ARM926EJ-Sid(wb) rev 5 (v5l)
BogoMIPS        : 99.73
Features        : swp half thumb fastmult edsp java
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant     : 0x0
CPU part        : 0x926
CPU revision    : 5
Cache type      : write-back
Cache clean     : cp15 c7 ops
Cache lockdown  : format C
Cache format    : Harvard
I size          : 32768
I assoc         : 4
I line length   : 32
I sets          : 256
D size          : 32768
D assoc         : 4
D line length   : 32
D sets          : 256


Hardware        : Oxsemi NAS
Revision        : 0000
Serial          : 0000000000000000


Groan, okay so I rebuilt ffmpeg without any of the CPU extensions because clearly this processor doesn’t have a single extension of which ffmpeg makes use.  I should be good to go right?

Wrong.

Moral of the story:

The ARM926EJ-Sid(wb) rev 5 (v5l) is so slow that conversion of 15 seconds of Audio and Video using ffmpeg took more than 5 minutes to complete.  Some people have said that the disk access is the bottleneck, but this simply isn’t the case.  vmstat(1) during conversion revealed that disk I/O was well underutilized.  The load average was well over 4 during conversion.  If you had the bright idea to use pyTivo on your Western Digital MyBook World Edition, think again.