Last year craigslist deployed MongoDB for its multi-billion document posting archive, largely due to its schema-free nature and built-in sharding and replica sets. Since then we’ve looked at it for other projects–specifically high-volume and multi-datacenter. In the process we’ve learned more about where other features do and don’t work so well, including replication, capped collections, and compound indexes. This presentation wil recap what’s worked well for us and discuss the other issues we ran into for new projects, as well as possible improvements in the the design or enhancements for MongoDB.
http://www.10gen.com/presentations/mongosf-2012/mongodb-at-craigslist-one-year-later
Last year craigslist deployed MongoDB for its multi-billion document posting archive, largely due to its schema-free nature and built-in sharding and replica sets. Since then we've looked at it for other projects--specifically high-volume and multi-datacenter. In the process we've learned more about where other features do and don't work so well, including replication, capped ...
It appears as though I never posted this video of our DEFCON 17 talk. Enjoy!
It appears as though I never posted this video of our DEFCON 17 talk. Enjoy!
The audio for our SXSW panel is now up. You can listen to it here.
This year at SXSW I met up with some hacker friends and we started theorizing about two-factor authentication under Linux. Darren, who hosts the show Hak5 on revision3, pulled out his camera and started filming the entire process of Eighty of Dual Core and I working through the problem. What’s interesting about this segment isn’t the proof-of-concept two-factor authentication daemon on Linux, but the process one goes through when designing a security service. Indeed, most of us in the hacking community would love nothing more than to communicate to the world that there is value in thinking about how people will attack your software.
There are certainly more formal ways to evaluate a software system’s design, and that is even more true for protocols. I think this video should show that you can have fun with the process, and that there is huge value in finding skilled interlocutors for your design and attack sessions. If there is one thing I’ve learned over the years: it’s that the loan-wolf design strategy always falls far short of the mark when you have a few good people’s heads in the problem. Most importantly, have fun with the process. Put on your white hat, and think up a good solution. Then toss on your black hat and tear it apart.
The two-factor authentication daemon, killallthehumansd, can be located on github. Enjoy the show:
This year at SXSW I met up with some hacker friends and we started theorizing about two-factor authentication under Linux. Darren, who hosts the show Hak5 on revision3, pulled out his camera and started filming the entire process of Eighty of Dual Core and I working through the problem. What's interesting about this segment ...
If you’re going to be at SXSW this year, then come to our panel talk with the EFF, SpiderOak, and Randy Cadenhead of COX Communications. The talk is entitled “Fighting for Your Users Without Becoming a Target“, and will be taking place:
here is the description:
The Internet is a fantastic resource for sharing and storing ideas, information, and creative works. But users — individuals and companies — can’t take advantage of that bounty without help from a network of large and small service providers, from social media services like Facebook to storage services such as DropBox and SpiderOak. Too often, these providers are cowed by legal threats into taking down perfectly legal material (like the Facebook page you use to network for your business) or revealing private information about their users. How can you earn your users’ loyalty by doing better, and how can you help ensure that the services on which you rely do right by you and your customers? What legal risks do you need to watch out for, and how can you make them go away? A group of experienced lawyers and business owners will help you answer these questions from a legal and practical perspective.
For more information on this panel, you visit the SXSW description page.
If you're going to be at SXSW this year, then come to our panel talk with the EFF, SpiderOak, and Randy Cadenhead of COX Communications. The talk is entitled "Fighting for Your Users Without Becoming a Target", and will be taking place: Saturday, March 10 3:30PM - 4:30PM Omni Downtown Capital Ballroom here is the description: The Internet is a ...
A little over a year ago Project DoD started building a whole new batch of servers, which we outfitted with commodity Seagate Barracuda disks. When we were benchmarking the speed of the new servers we noticed a major disk performance problem: rotational vibration. For more details see the original post. At first we thought it was the design of the 1U server case, but we soon came to figure out that commodity drives weren’t as good as they were back in 2003. As capacity increased, the inner workings of platter disks became much more sensitive to outside influences. These influences are usually vibrations, and they are very hard to avoid in a server environment (fans, other disks, other servers in the rack, higher levels of ambient noise).
Well, a few months ago, both disks in one of our new servers failed on us. We took the opportunity to purchase two new Seagate Constellation ES 1TB 7200 RPM SATA 6.0Gb/s 3.5 drives, as they were advertised as near enterprise class drives. At the same price we paid for the Barracuda disks a little over a year ago, it was hard to argue with the value in these new Constellations. Now, granted, they are not enterprise drives, but how would they perform when stacked up against the old disks in the same server hardware? Well here is the answer:
Next round, we hope that we can start to get our hands on mass storage Solid State Disks, but it all depends on the price when you’re a broke nonprofit spanging for bitcoins on the Interwebs.
12ZBRTvEz4z8Z85iZEdt2kRiRsSaQAVPVE
or
Make a donation to Project DoD.
A little over a year ago Project DoD started building a whole new batch of servers, which we outfitted with commodity Seagate Barracuda disks. When we were benchmarking the speed of the new servers we noticed a major disk performance problem: rotational vibration. For more details see the original post. At first we thought ...
We just bought two new Seagate Constellation ES 1TB 7200 RPM SATA 6.0Gb/s 3.5 drives to build a new MySQL box. That’s right, we are getting ready to upgrade MySQL (to 5.5) on dod.net, but downtime should be low. We are going to lock out adding new databases in the account manager, then slowly migrate each site individually to the new MySQL.
As it turns out, we are actually buying the new drives because they perform a little better under rotational vibration. What’s more, one of our backup servers has had a double drive failure, and we need to replace both of those crap Seagate Barracudas. So we are taking two Barracudas out of our new MySQL server, and replacing them with the more performant Constellations, then shipping the Barracudas off to our backup server (so they can fail again). Sometimes I wish dod.net had more money for quality hardware.
Anyway, that is the plan. All of you who’ve been begging for a MySQL upgrade, the time is almost here.
We just bought two new Seagate Constellation ES 1TB 7200 RPM SATA 6.0Gb/s 3.5 drives to build a new MySQL box. That's right, we are getting ready to upgrade MySQL (to 5.5) on dod.net, but downtime should be low. We are going to lock out adding new databases in the account manager, then slowly ...
Recently, we needed to build a reverse caching proxy to take some load off of one of our servers. I’ve been playing with Node.js for about a month, and it seemed like just the thing to implement something quick in. For those of you that don’t know about Node yet, it is a JavaScript implementation of an event based server framework that runs on Google’s V8 JavaScript Engine. Being primarily a C programmer, I was very skeptical at first, but I had recently started contributing to the Haraka mail server, and I knew Node.js was so easy to pick up and start coding in. I know there are other canned reverse caching proxy solutions out there, but some were not event based servers, and others looked hard to customize. In the end I made a calculation that I rarely do: to roll our own reverse caching proxy because it looks like more fun.
The first thing I did was look to see if anyone had written a reverse caching proxy in Node.js. Unfortunately, no one had done the caching part, but I did find something called node-http-proxy. And even though it did not do caching, or appear to have any intention of adding it in the future, it looked like a great project to fork and start hacking on. So I forked node-http-proxy, and within about five hours one evening after work, I made all the changes needed for a very basic, and slightly buggy, reverse caching proxy. Nevertheless, this was good enough to throw into production, and it saved our server for what was to become about a week of Apache crushing traffic.
Over labor day weekend I made a bunch of bug fixes that helped the reverse caching proxy play nice with WordPress, and in the end we’ve wound up with a reasonably stable reverse caching proxy for dod.net users. Please watch the video below for a better explanation of how it works.
We can already see that our reverse caching proxy is going to be a huge help on our censorship resistant hosting project. In the past we’ve avoided using Tor hidden services because the Tor network is a little too slow for our user’s needs. With a reverse caching proxy, Tor hidden service hosting me be viable for dod.net users. We are eager to test this theory. As for now, the reverse caching proxy still needs some work. I will post updates as we add new features. Here are just some of the things we have left to do:
Stay tuned for more details.
Recently, we needed to build a reverse caching proxy to take some load off of one of our servers. I've been playing with Node.js for about a month, and it seemed like just the thing to implement something quick in. For those of you that don't know about Node yet, it is a JavaScript ...
Here is the content of the DMCA talk Tiffany Rad and Christopher Mooney gave at H.O.P.E. and Blackhat in 2010. The video starts washed out, but clears up shortly after. You can find the whitepaper for the talk at http://www.dod.net/dmca_paper.pdf, as well as the slides for the talk at http://www.dod.net/dmca_slides.pdf.
From now until June 1st any donations dod.net receives up to $2000 will be matched times four by a few anonymous donors. That is, if you donate $100, on June 1st we will get a $400.00 matching donation, bringing the total to $500.
This very generous offer means that if we can get $2000 total donations, we will receive $8000 matching, thus making for $10,000 total in donations.
This offer comes because we are already a few months behind in donation collection, and by the end of the month we need to cover our Oakland server expenses. So if you were thinking of donating, now is the time to make even small amounts worth it. At very least, please spread the word.
Please go to our donation page and contribute today!
From now until June 1st any donations dod.net receives up to $2000 will be matched times four by a few anonymous donors. That is, if you donate $100, on June 1st we will get a $400.00 matching donation, bringing the total to $500. This very generous offer means that if we can get $2000 total ...
