Saturday, February 21, 2009
A hotspot logging format
This logging format is intended to be a compact notation usable for equipment and software implementing wireless hotspots (Wi-Fi) allowing remote user access, DHCP servers, etc, to keep long-term records of addresses assigned.
The goal is to be a lightweight compact format, suitable for keeping the info in archives for 2 years as may be required in the US by pending legislation.
The format should be a binary format, as compact as possible.
Each record should be of a fairly uniform size, but future extensibility should be provided for.
The following is a 256-bit record format.
All integers are represented in the network byte order.
Values are padded up to the required length with NUL bits inserted to the right.
The first 4 bits are used to indicate record type and version. In a standard record, they should be all set to 0 presently.
The next 4 bits are reserved for future use, and always set to 0, until future versions.
The next 4 bits indicate "action", (i.e. Lease = 0000, Un-lease = 0001)
The next 4 bits are reserved for a "hardware address type", all bits are 0 for Ethernet
The next 4 bits are reserved for "assigned address type", use the IANA address family number
The next 48 bits are for a "hardware address".
The next 33 bits are the "action" timestamp, which is an offset in seconds since the UNIX epoch, Jan 1, 1970 00:00:00
The next 24 bits are for "longevity" in minutes, i.e. DHCP lease duration. An infinite lease is recorded as all bits "1".
The next 128 bits are for "assigned address data"
The next 3 bits are a simple checksum
By recording events using a binary logging format, the storage requirements are drastically reduced over text-based logging formats.
Automated searching and archiving also becomes much easier.
The number of records that need to be generated can be minimized in a home network environment by using a fairly long lease duration, say one week.
A DHCP server can retain a hash table in memory and on disk that points Ethernet hardware addresses to the proper log position for the past lease.
When a lease is requested, the hash table is first searched for the hardware address.
If found, the existing log entry is loaded, and the lease entry is extended, instead of generating a new identity log, the assigned address is the same.
However, the hash table entry must be purged if a lease is allowed to expire and a new lease assigns the IP address to a different hardware address.
Also, if the host releases the lease, or goes offline, an 'unlease' record should be generated, or the lease duration should be shortened to indicate the proper duration according to a system clock reliably sync'ed to a trusted time source.
Renewals are effectively the same as increase in the original lease duration.
Some applications might like to record a new entry altogether, even for such lease extensions, and a log reader application must be prepared to accept either behavior.
Shorter record formats might be defined to indicate inactivity of a hardware address.
In a home environment, there are fewer hardware devices, and a hash table and lease-duration update strategy makes sense, and allows for a minimal amount of storage to record the identity assignment history of all IP addresses with no loss of data.
In a public hotspot, there are many hardware devices, and it is more optimal to simply use an archival log and plenty of storage.
Still, the storage requirements are not huge.
100,000 full-sized event records can fit within 3 megabytes of storage.
For a 2 year retention period, this allows for 136 address assignment events per day,
or 5 per hour.
In a home environment, this is ample, particularly if long lease durations are used, so addresses rarely need to be assigned, and with IPv6 on the LAN, the lease assignments can be permanent with no danger of subnet exhaustion.
Especially, when we consider a variety of compression schemes can be used to further reduce storage requirements. ¶ 2:54 PM
The goal is to be a lightweight compact format, suitable for keeping the info in archives for 2 years as may be required in the US by pending legislation.
The format should be a binary format, as compact as possible.
Each record should be of a fairly uniform size, but future extensibility should be provided for.
The following is a 256-bit record format.
All integers are represented in the network byte order.
Values are padded up to the required length with NUL bits inserted to the right.
The first 4 bits are used to indicate record type and version. In a standard record, they should be all set to 0 presently.
The next 4 bits are reserved for future use, and always set to 0, until future versions.
The next 4 bits indicate "action", (i.e. Lease = 0000, Un-lease = 0001)
The next 4 bits are reserved for a "hardware address type", all bits are 0 for Ethernet
The next 4 bits are reserved for "assigned address type", use the IANA address family number
The next 48 bits are for a "hardware address".
The next 33 bits are the "action" timestamp, which is an offset in seconds since the UNIX epoch, Jan 1, 1970 00:00:00
The next 24 bits are for "longevity" in minutes, i.e. DHCP lease duration. An infinite lease is recorded as all bits "1".
The next 128 bits are for "assigned address data"
The next 3 bits are a simple checksum
By recording events using a binary logging format, the storage requirements are drastically reduced over text-based logging formats.
Automated searching and archiving also becomes much easier.
The number of records that need to be generated can be minimized in a home network environment by using a fairly long lease duration, say one week.
A DHCP server can retain a hash table in memory and on disk that points Ethernet hardware addresses to the proper log position for the past lease.
When a lease is requested, the hash table is first searched for the hardware address.
If found, the existing log entry is loaded, and the lease entry is extended, instead of generating a new identity log, the assigned address is the same.
However, the hash table entry must be purged if a lease is allowed to expire and a new lease assigns the IP address to a different hardware address.
Also, if the host releases the lease, or goes offline, an 'unlease' record should be generated, or the lease duration should be shortened to indicate the proper duration according to a system clock reliably sync'ed to a trusted time source.
Renewals are effectively the same as increase in the original lease duration.
Some applications might like to record a new entry altogether, even for such lease extensions, and a log reader application must be prepared to accept either behavior.
Shorter record formats might be defined to indicate inactivity of a hardware address.
In a home environment, there are fewer hardware devices, and a hash table and lease-duration update strategy makes sense, and allows for a minimal amount of storage to record the identity assignment history of all IP addresses with no loss of data.
In a public hotspot, there are many hardware devices, and it is more optimal to simply use an archival log and plenty of storage.
Still, the storage requirements are not huge.
100,000 full-sized event records can fit within 3 megabytes of storage.
For a 2 year retention period, this allows for 136 address assignment events per day,
or 5 per hour.
In a home environment, this is ample, particularly if long lease durations are used, so addresses rarely need to be assigned, and with IPv6 on the LAN, the lease assignments can be permanent with no danger of subnet exhaustion.
Especially, when we consider a variety of compression schemes can be used to further reduce storage requirements. ¶ 2:54 PM
