PUBLIC-NOTICES: Forum Changes/Fixes. Any Questions?

[TheCotMan]

The failure to load avatars was actually 2 problems. The new forum software does not ship with the avatars we use, so I imported those from backup. Second, the local OS's library for JPEG support was inadequate, so I built a new one from source, and then built PHP against it, allowing the forums to generate thumbnails of JPEG images. (GIF was working fine, which is why some avatar thumbnails worked fine.) After upgrade, I bounced service, and re-generated all thumbnails and sig images. Some thumbnails will show up as "blank" because of browser caching. To test this, I loaded Nikita's thumbnail image, which showed me the blank image, but a "force reload and ignore cache" in browser caused the generated thumbnail to appear as it should.

If your avatar still only shows up in some views but not others 24 hours after this post, please report this as a problem so I can investigate a possible third or fourth problem. You can try to force-reload of all page, bypassing your local cache in your web browser, too, and see if that shows you the fix. In Firefox, "Control-Shift-R" from keyboard.

Thanks!

Please report other issues you find here or in the other thread: https://forum.defcon.org/forum/gener...ums-discussion

Thanks!
-Cot

[VideoPod]

Ok Cot,
I'll post just to see what my avatar looks like...

[TheCotMan]

Posting is kind of broken with new security policies. When you post, you see an error page about node-id and no indication that the post working, even though it succeeds. We know about this and few issues related to features with "hover over" to enable. We'll investigate these and find a fix. Sorry about the degraded service.

[VideoPod]

Just Testing it out CotMan....

[TheCotMan]

Most of the features we had prior to the policy change have been restored, but more work is needed to help our users and their clients with good policy information.

For testing, feel free to use this thread for testing: https://forum.defcon.org/forum/rules...rums-and-posts

[TheCotMan]

There is a bug in the Firewall that causes an attempt to create a new thread to lead to your IP being blackholed for a while. Your new thread actually happens, but until the firewall is no longer angry at your IP, you can't see posts from that IP. It is a known problem and Jeff is looking into it. (Also, yes, this also blocks me when I create new threads/topic, but not when I reply to existing threads.)

[TheCotMan]

I have an update from Hackajar on Contests/Events approved for allocated resources (space, power, tables, chars, support) for our next DEF CON.

I am planning to add new forums for these contests tonight.

[TheCotMan]

For a few weeks, I've been without shell access to the forums. I contacted the firewall admin to work on a resolution, so I could get shell access and complete some upgrades. There were plans to address firewall issues over last weekend. When access to the forums was lost on Monday, I assumed it was part of the firewall work scheduled for the weekend.

It was not.

After more than 10 hours, I began to suspect there were problems with the maintenance cycle, so I asked the firewall admin when access to forums might be restored, and was informed there was no maintenance being conducted. The server was having problems. A third person was asked to inspect the server, and found it in a terrible state. It was rebooted, we repaired several items and started looking for causes.

Everyone was off-site. Wed morning, someone was available to visit the server in person. Service was restored Wed morning.

I've been really busy at my real job for the last 2 weeks, so it has been hard to get time to work on the forums to diagnose problems and plan fixes.

I found several possible causes, and of those that I found which could be addressed, time had to be spent to work on those. One of these will take weeks of review to better tune it.

DB repair was completed Wed. night.

I extended the down-time to complete other maintenance and upgrades, which I was not able to repair without remote access.

Upgrades and other changes to system and service configs were completed Thursday night.

Now, we are "early Friday morning" a little before 1am pacific time, and service is being restored. I am using this as a test post to diagnose fixes of other reported problems.


Finally, after all of this, service is restored, and I once again have shell access, so the back-log of work I ws supposed to do with new forums for contests and events last weekend can proceed this weekend.

Apologies for the delays. Now that access is restored, I will be creating forums for contests, events and social gathers as approved by Contests & Events goon, Hackajar, this weekend.

Sorry for the down-time and delays.

[TheCotMan]

The last 7 days have been terrible.

Last night, we ran out of disk space on the server that holds the forum DB. Soo stupid.

All posts that actually were completed without error, and posted before the DB crashed when we ran out of disk space were saved. Any post attempted at the time of the crash was lost. (If the forums did not present you with a failure on posting, then we should have your posts and threads restored. If you saw an error at the time of posting, then your post or thread probably never made it to disk -- out of space.)

I have purged the old DB, and data files used by the DB, re-created the DB data dirs, init-ed the DB to use those data dirs, recreated the all the schema used by all services including forums, then restored all DB and tables from backups. There was not enough space to preserve the old data files and install the new ones. Restoration worked, and we are live again.

It is a bit extreme to go that far to restore, but once you run out of space on a DB server, underlying files can be damaged in ways that make diagnosis of trouble difficult to detect and resolve. Best solution for long-term support if burn it all down, and start over and restore. We did this.

Sorry about that,
-Cot

[The Dark Tangent]

Slow forums, and other changes

As you have noticed the new software version is nicer, but it is slower. When we switched to the new version in anticipation of this we also switched to a faster server. The old one was a low power 1.8 gig 2 core 64bit cpu with 8gigs of 800mhz ram with SSD drives. We switched to what it is running on now which is also lower power but 2.4 ghz 8 core 64bit cpu with 32 gigs of 1333mhz ram and ZFS mirrored SSD drives - quite a bit faster! But this new software beat us down so it seems even slower than the old stuff. No amount of tweaking by Cot or myself has been able to speed it back up.

So

We are going to migrate yet again, this time to a high power server. If it doesn't run fast on the new system it just won't run fast at all. It will be a 4Ghz Xenon E5 6 core/12 threads 32 gigs of 1866mhz ECC ram with SATA3 mirrored enterprise SSD disks.

This should happen in less than two weeks.

Also a reminder that there is no longer a wait from when you sign up to when you can post, so get your friends on here and start planning. I'm really looking forward to the faster system.

Finally, once the faster system is up we should be ready to start playing with a mobile app we licenses for Android and iOS that will allow you to be much more mobile with ease. Something we hope to take advantage of during the con if testing works out.

More soon!

[L0g1c]

We are going to migrate yet again, this time to a high power server. If it doesn't run fast on the new system it just won't run fast at all. It will be a 4Ghz Xenon E5 6 core/12 threads 32 gigs of 1866mhz ECC ram with SATA3 mirrored enterprise SSD disks.

What seems to be the bottleneck in speed? CPU/RAM/DiskIO?

You seem to be hinting at DiskIO but given the upgrade I'm unsure. How much storage are we looking at? Perhaps SATA drives should be abandon in favor of PCIE SSD drives? (SATA III bus is 40-50x slower than a PCIE x16 bus).

~L0g1c

[The Dark Tangent]

It's the new version doing DB lookups and more renderings and so on. The core software is just bigger and slower, despite optimizations, so the only solution seems to be more megahertz, we tried for more cores and that didn't do it.

The Xenon + faster memory should be really visible. Recompiling world on the 2.4gig 8 core Atom vs. the 4gig 6 core E5 is crazy, about 6 times or more faster.

[L0g1c]

It's the new version doing DB lookups and more renderings and so on. The core software is just bigger and slower, despite optimizations, so the only solution seems to be more megahertz, we tried for more cores and that didn't do it.

The Xenon + faster memory should be really visible. Recompiling world on the 2.4gig 8 core Atom vs. the 4gig 6 core E5 is crazy, about 6 times or more faster.

In that case I can't recommend the PCIE method of SSD storage enough. If overall storage capacity isn't feasible on PCIE-SSD cards then you can do SATA III hybrid drives for the bulk of the storage + PCIE-SSD card using software such as: EnhanceIO, Flashcache, or Bcache.

~L0g1c

[The Dark Tangent]

We're currently running ZFS mirrored Intel enterprise SATA III SSD drives, and with the 32gig of ram all reads / writes are happening out of ZFS ram cache already before hitting the SSDs. That's why I am feeling so certain it's cpu crunching with PHP and not Disk I/O.

[TheCotMan]

The Dark Tangent replaced the old hardware with the new hardware today, upgraded stuff, and this is a test post after I rebuilt our services from source, and upgraded/migrated the DB.

Will it post?

Forums should be up and reachable by regular users soon.