Holiday in the UK!

Well hello all! I have been fairly quiet over the past few weeks, mostly because I’ve been on holiday! I’ve been in the UK for the past two weeks spending time with my dearest friends in the lovely town of Milton Keynes.

I won’t be posting much about this today, mostly because I think this visit will be better told in pictures. I’ll get around to posting those in the Photos section once I’m back in the states.

For now, at least, a pool match beckons, so more will follow later! Bye!

A Long Time with no Update // Story of my life for the past 6 months

Hello blog my old friend. I’ve come to talk with you again.


Job and Apartment Changes

I haven’t actually updated this blog since March, back when I reported that my site was compromised with various PHP shells. A lot of things have changed then, and while I have kept up with various people over Twitter and Facebook and in person, I haven’t really taken the time or the effort to record my goings on here. There are a multitude of reasons for this, but they can generally be summed up as follows:

  1. Laziness.
  2. Fear – yes, fear. I’ve been feeling a lot of things for months now that I didn’t want people to know.
  3. Respect for other people’s privacy.

But a great, great deal has changed in six months, so much so that I finally let some folks go, and I’m going to talk about a few of the things I’ve been afraid to discuss here in public.

I guess I’ll discuss first various big life changes I’ve gone through over the last six months. I now live alone. One of my roommates and I had serious disagreements in the second half of last year that badly damaged our trust in one another. It was very traumatic for both of us. We’d previously been very close friends, and after our disagreements and arguments it just wasn’t quite the same. We mutually agreed that I needed to live on my own, and I resigned from the shared lease we were on and got my own apartment further in Austin.

Since then, we’ve stopped speaking. We’re no longer friends on Facebook, Twitter, or Skype, or elsewhere, and honestly I’m ok with this. I’ve decided to let him go. I have various reasons for deciding that, but it’s simply that neither he nor I have expended any serious effort to stay in touch and after what we went through I don’t think that it will do either of us any good to try. Even now, I’m a little afraid that he will eventually find his way here and read this, but you know what, that’s ok. Mayhaps that will be a little closure for him, as writing this is for me in a way. Mayhaps he will feel betrayed and upset that I didn’t actually come to tell him this myself. I’ll understand that, but to be honest I want to move on with my life and enter this new chapter that I’m suddenly finding myself in.

Speaking of, I may or may not have mentioned this in one of my other blog posts. I have a new job! Well, not really new; I’ve been working at IPsoft for a year this September, but basically, I don’t work at HostGator anymore and I’ve transitioned over to “grown up” system administration at a much bigger company that is right in the center of the action in the IT industry.

I am a Linux Platform Engineer, doing what I have grown to love doing. Administering systems, fixing problems, being technical, and generally being amazing. It has been everything that I had hoped for in a change from web hosting to something bigger and more important, and it has been a huge boon to my personal and professional development. I have a boss who coaches and looks out for me, and I have a product and a service that I can believe in and provide service for. I can’t really ask for much more than that.

Changes, and Why it Terrifies Me…

Alas, the changes started one year ago are still ongoing. It wasn’t just switching jobs and then apartments. It’s been a coming of age experience.. yeah, another one (college, Korea, Austin, HostGator… those all used to be my coming of age experiences). It’s been an explosion of knowledge, exposure, trust, freedom, integrity, and responsibility. Put simply, I’ve entered a whole new world.

And it all terribly frightens me. This is a world I never really knew existed, and it’s starting to make sense to me. Politics, leadership, decision making, projects… people… I’m only just beginning to see it, and it’s both fascinating and terrific. It’s like… all this time I have been living in a layer below, which is in itself shocking. College, Korea, Jiffy Lube, HostGator. Despite all those experiences, it feels as though I have only just now broken through this layer below into a layer above, and doing so I begin to understand what people go through in adulthood and why they start believing what they believe. I struggle to try and keep who I am.

I am doing extremely well. So far, I’ve been very successful and I’ve been very happy with how I have acquitted myself. And I have received the respect and admiration of almost everyone I work with. And in this, I start to feel a certain sense of destiny… or doom. I feel very proud of myself, but also have a feeling of trepidation for what’s coming, like something big is in my future and I don’t know what it is. I just have to learn everything I can now so that I can deal with it when it comes.

A lot of this is probably frustratingly vague. That’s intentional. Satisfy yourself with this basic summary:

I feel my very character being tested very strongly at the moment. It frightens me, and I pray that I will pass that test.

Bernie Sanders, 2016, Optimism, and the Future of American Politics

It’s interesting that I’m feeling that doom on myself and, at the same time, we’re dealing with the very beginnings of the 2016 Presidential primaries. And my, what a circus it’s been – mostly on the Republican side. Donald Trump scares me. I’ll talk about him some other time though.

Bernie Sanders is what I really wanna talk about, at least for a little bit.

You should know that I studied political science in college, transitioned from Republican to Democrat in my last year of college, voted for Obama in 2008, and then promptly moved off to South Korea in 2009 after graduating. From that point, I was depressed on the state of American politics and governance and have been ever since. South Korea, by all standards I could apply to it, was a far better country and governed itself far more effectively than we – our American exceptional selves – seemed capable of imagining.

Korea had inexpensive and very good medicine, affordable food, a culture that worshiped a good education, the best rail system I’ve ever ridden and a world class infrastructure, yadda yadda. At this point, my experience catches up with my knowledge and current world view.

Continuing, I lived in Korea from 2009 to 2011. I moved back to Texas that year. I was as depressed over the state of American governance and politics as ever. I knew what I believed in and what I felt was right for the country, but there was no point talking about it or debating it. It would come to naught. The Idiot Right would shout it down and the “Mindless Middle” (as some say) would follow along cuz it sounded better. I didn’t belong here. My stake would eventually be planted elsewhere, so why care about what was happening in my own country?

And then 2015 shows up. Despite losing all of Congress in 2014 in a swell tide of conservatism and dissatisfaction with government, suddenly the American Left sees victory after victory. The ACA wins it’s second judicial challenge in the Supreme Court, and gay marriage is legalized everywhere by a court ruling just a few days later. The country celebrates! A few start complaining, and they are widely condemned by everyone.

And then, Bernie Sanders came forward to run for President, the one candidate who represented anything close to the kind of policies that I would dare dream for – some of the sensible social policies that I see out of Germany or Korea or Scandinavia, or at least the willingness to debate them and call it what it is instead of Crying Wolf (that is, SOCIALISM!!! or COMMUNISM!!!). And, to my complete astonishment, he starts drawing incredibly huge crowds and getting lots of attention! I felt emboldened, happy, optimistic, like the country wasn’t doomed to a perpetual state of deadlock and dysfunction. If thousands and thousands of folks would turn out to see a self-proclaimed socialist speaking optimistic things about a US where economic justice was a thing, where the state supported it’s students and poor folks, where we could again lead the world into a scary future instead of existing as a static, chaotic mishmash of opposing, state applied policies, then… maybe… just maybe, this country was a thing I could put my stake into. Maybe I could give a shit again.

Now, that’s by no means a complete statement on all of the things that Bernie promotes. I realize that his path to the Presidency is an incredible long shot. Fivethirtyeight.com puts his odds at about 15% right now compared to Hillary Clinton’s 85%, mostly because he’s not drawing any minority support in the Democratic party. That’s all fine. I support him anyway, and here are my reasons:

  1. You should always vote your conscience in the primary. That vote, at least, is not quite so meaningless as the general election. 😛
  2. Bernie is doing something for the American Left that I think Barry Goldwater did for the American Right back in the 1960s. Back then, the Left was ascendant. The FDR coalition had held for 30 years. And many conservatives who worked for his campaign and voted for him were immensely influenced by his candidacy, message, and methods. This butterfly wing flap didn’t really develop into a hurricane until Ronald Reagan’s election in 1980, but that’s where it really got started if history has anything to say about it.Point is, I taste something very similar happening with Sanders now. That significant portion of young people who are emboldened by his campaign to be left and proud are going to become a passionate force in American politics 20 years from now; mark my words. And it will be good, at least only if we can start talking about some of the sensible solutions the Left offers to our country instead of flinching in fear of the long dead Communism bogieman.

The same can also be said for Trump – who is currently leading a Know-Nothing Party revival – and the 20% of Republican voters that he is beating up into a froth. I weep for the state of our national politics if this sort of discourse that we’re going to have to have about immigration, and I know that this portion of the Right are also going to be a huge force in American politics for years to come.

Family, Death, Friends

My grandfather on my mother’s side died on August 17th. I attended his funeral up in Dallas yesterday. It was a very impressive event and I learned a few things about him that I’d not known before. Big Jim, as we called him, was in the Air Force and served in the Korean war, for instance.

The military presented him with a three gun salute and played taps for him, and the Dallas PD (for which he worked for 27 years, I think) were also present to give their respects. As for Big Jim himself, he was cremated and laid with us in an urn, to which we all faced. It struck me that the man I’d loved as my dear grandfather was now confined to such a small container.

After the ceremony, we went to have lunch with local family, and after that I went to visit my grandmother, Big Jim’s widow, Nana. I was profoundly affected during that visit, such was her state and how different it was from what I remember. It was very hard. 🙁

I was very, very glad that I got a chance to see her and give my love to her, though. I was touched at how happy our visit made her and how she was still able to express her love. I’m going to remember that forever…

Written while listening to

http://www.youtube.com/watch?v=6UlTJR9FY8Y

My Site Got Compromised!

So, read the title! Someone managed to exploit my site and install some malware. Ok, ok.. not something that I’d want to be sharing publicly, you’d think, but I disagree. Granted, I might not be proud of the fact that my site had exploitable code in it, but I also think that it’s important to tell folks – who host sites as a hobby or for a living – that it’s possible for things like this to happen. And so, now I share with you just how my site got compromised, and how I found out about it, and how I dealt with it.

Today I was doing some maintenance on my server cluster. I was installing some security software on one of my auxiliary servers and I was testing out a scan script.

Now, a contextual thing to keep in mind is that this particular auxiliary server is my failover for all of the sites I host. If I were expecting my primary content server to go down for whatever reason, I can failover to this particular secondary just by running a simple script. I have scheduled cron jobs that run every day, and they sync up all of the content that I have on my primary content server to this secondary server so that everything matches.

Anyway, I was testing out my scan script. Lo and behold, the scan turns up a single malicious file that ought not to be there. My security admin training from HostGator suddenly kicks in, and I check out the file and look at its contents. Sure enough, it’s malicious! Well! If the file is on my secondary server, it has to be on the primary server from which it’s synced, so I went over there and sure enough, there it was. I looked through my scan logs and found evidence there as well.

So, my site got compromised. Now it’s time where I go over how I dealt with it. This is a bit advanced, I think, and I’m not tailoring this precisely for new users, but it might help some folks to find their way if they need to.


First thing that you do in a situation like this, is you run the ‘stat’ command on the malicious file that your scanner found. I did this, and here is what I got.

File: `ajax-upload.php’
Size: 177962 Blocks: 352 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 134650 Links: 1
Access: (0644/-rw-r–r–) Uid: ( 500/ nope) Gid: ( 500/ nope)
Access: 2015-03-22 00:05:54.445132850 -0500
Modify: 2014-10-16 13:03:52.000000000 -0500
Change: 2015-02-04 21:10:28.178217693 -0600

What does this output say? It says that this file was last Accessed on the web or on the local system at 2015-03-22 00:05:54.445132850 -0500This is the month, day, year, second, and timezone of the server (-0500 is 5 hours behind GMT). It also says that the file was last Modified on 2014-10-16 13:03:52.000000000 -0500, and last Changed on 2015-02-04 21:10:28.178217693 -0600.

What’s the difference between Modify and Change? Aren’t they the same? Well, yes and no. In this particular context, Modify refers to the last time that the file contents were changed, and Change refers to the last time that the meta data of the file changed (permissions, location, and such). So, essentially, the last time that the file contents were modified was on October 16, 2014, long before it ever made it onto my system. That’s probably when the programmer last modified the file before some script kiddy decided to make it a part of their h4ck0r bot. The Change date is what we want to pay attention to then, since this indicates when this file’s metadata (on my system) last changed.

I go through and, using the change date as a reference, look through my http access logs. I find the following:

91.217.82.160 – – [04/Feb/2015:21:10:19 -0600] “GET /wp-admin/ajax-upload.php HTTP/1.0” 404 2514 “http://google.us” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
91.217.82.160 – – [04/Feb/2015:21:10:24 -0600] “POST /wp-admin/admin-ajax.php?action=settings_upload&page=pagelines&pageaction=import&imported=true HTTP/1.0” 200 27 “https://prannon.net” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
91.217.82.160 – – [04/Feb/2015:21:10:26 -0600] “GET /wp-admin/ajax-upload.php HTTP/1.0” 200 114 “http://google.us” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”
91.217.82.160 – – [04/Feb/2015:21:10:27 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 2654 “http://google.us” “Mozilla/4.0 (compatible; MSIE 8.0.1; Windows NT 6.1)”
91.217.82.160 – – [04/Feb/2015:21:10:28 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 3486 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:29 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 5522 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:33 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 16233 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:46 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 23382 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”
91.217.82.160 – – [04/Feb/2015:21:10:48 -0600] “POST /wp-admin/ajax-upload.php HTTP/1.0” 200 3583 “http://google.us” “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”

Note how the time lines up in these log entries and the Change time in the ‘stat’ that I ran above? Since the logs line up with the time stamps, it means that this is where the file came from. If I follow this set of logs back to the original spot, I can see exactly what happened.

91.217.82.160 – – [04/Feb/2015:21:10:19 -0600] “GET /wp-admin/ajax-upload.php HTTP/1.0” 404 2514 “http://google.us” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”

The malicious user checked to see if their shell existed. It doesn’t. If you look, you can see the 404 “Not Found” error code in this log entry.

91.217.82.160 – – [04/Feb/2015:21:10:24 -0600] “POST /wp-admin/admin-ajax.php?action=settings_upload&page=pagelines&pageaction=import&imported=true HTTP/1.0″ 200 27 “https://prannon.net” “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0”

Seeing that their shell didn’t exist, they used a newly published theme exploit against the theme that I use on my site. I actually found some links for this exploit that pretty much confirmed that this is exactly where my malware came from.

http://www.pagelines.com/themes-requiring-updates/

http://blog.sucuri.net/2015/01/security-advisory-vulnerabilities-in-pagelinesplatform-theme-for-wordpress.html

And now I know where the malware came from and how my site was exploited. I can take action. I immediately updated my theme, and then looked through my entire site directory structure for any and all malware that might have been left behind. I used a ‘find’ command to do it.

find -type f -ctime +1 -ctime -90

Since I know when the original compromise happened, and since I just updated my site, I can create a narrow time frame during which the malicious user uploaded or modified files to contain their malicious code. Finding these, I can delete the files or I can clean them of bad code.

And then finally, I took action to make sure that my wp-admin folder was restricted to unknown users. In the past, I’d had .htaccess restrictions in place that were supposed to prevent unknown users from being able to access these sites. If this were working, it would have prevented this sort of compromise from ever happening. Remember from the logs above, the malicious user used a file within wp-admin to perform their exploit.

At some point, though, these restrictions stopped working properly, and… well here we are. Since my .htaccess restrictions aren’t working anymore, I enacted the restrictions at my front end layer instead, nginx. I’m weird! I use two http servers to do my bidding. I have nginx as a front end and apache as a back end. It’s sad that .htaccess doesn’t work 100%, but I can live with having the restriction in place at the nginx layer. Ultimately, however the site is secured, I’m happy.


So, that’s the tale of my rather sudden and random malware event of the day. Hopefully it’s somewhat interesting. Anyone is free to hit me up for questions on this point, and I’ll be glad to help out!

Wherein Jesse Rambles about Stuff that Doesn’t Really Matter

Hello again my long lost blog! How are you?

This is one of those rambling blogs that’s going to be something of a stream of consciousness. Normally, when I’m having a stream of consciousness moment, I run off to my twitter where I spooge all of my thoughts across short microblog tweets. That can be satisfying, kind of? Eh, not really. I’ve concluded that twitter is a horribad platform on which to methodically ramble on about stuff that isn’t important.

So what am I going to talk about in this particular blog post? What to ramble about? Well, to be honest, I’ve had this interesting situation on my mind. A few weeks or so ago, I started making a new friend on the twitter space. We interacted with some regularity and it was pretty good! I actually do enjoy some friendly and occasional casual interaction (I consider twitter to be casual enough). Sometimes? I guess? It depends on the person, as I’ve been reminded over the past few days.

Anywho, friend and I would interact a bit, and a about a week said friend added me to skype. Ok. I have no problem with that. I don’t really have some sort of off-limits rule with my skype list, and if someone wants to be on it I’ll generally add them and give it a shot. That is, if they want to talk to me more regularly, I’ll see how it goes and see if it develops well. If it doesn’t go well, I can always just remove them, right?

So the first conversation goes well! We talk about what we were doing in that moment and talked about jobs and beer and stuff. Harmless stuff, pretty much. Second conversation got off to an awkward start though, and that kinda set the tone of the rest of the conversations to me.

“Awkward in what way, Jesse?”

I’ll be honest, and I’ll say that it’s probably not entirely justified how I judged this, but it was basically a case of the fellow noticing me online, and saying that I ought to have said something. I made a joke, mentioning that I had only gotten off work a little while ago and asking for forgiveness if I didn’t turn to skype first thing. I was actually in the middle of getting settled in at a local cafe to do some drawing, so really and truly I wasn’t about to get on skype and start chatting with folks. Apparently my joke was out of place, though? Cuz friend sorta panics a bit and thinks I’m being horribly defensive, which caught me off guard.

And here’s the weird part. I tell him to relax. He then says that he was about to tell me to do the same thing, and that I was being very defensive. Umm… ok? And then the conversation goes well for the rest of the night and all is well until I head home.

That kinda set the tone for the rest of the short time that we were friends. We had a few more good conversations, but I started to feel like I was being pressured into conversations. It didn’t really help that I’d been having a few really tiring, icky days (blame me for staying up too late). But in any case, I spent some time feeling like he was freaking out over little things, and as I came to learn, he spent time thinking that I was freaking out and being defensive. This didn’t really make sense to me in the context I was talking to him in. It actually got really uncomfortable after a bit, and so one night I took some time to confront him on it.

I tried to be really nice and honest, but the conversation went very poorly and escalated very quickly. Needless to say, it felt like a nasty argument. Points exchanged were that I felt like he expected too much from me too quickly and that he was a bit overbearing in trying to converse. His points were that he was just trying to be friendly and that I was being inappropriately defensive in response.

But honestly, looking over the conversation, there was some silly stuff in there. Him accusing me of not being interested in anything that he was doing. Him judging me to be a “popufur,” as they’re called, due to my attitude regarding my skype list? It was one of the weirdest things I’d ever seen someone call me. And I said some silly stuff too, like why did I have to talk to him in particular if I was in any particular mood to chat with folks? I guess point is, we both rubbed each other the completely wrong way.

After talking for a little more than an hour, I got tired and frustrated and we called it off for the night. I spent a lot of time after that being pretty upset and actually kinda angry about the whole thing, and considering that I’d only been talking with this guy over skype for a week, I made an executive decision to just cut it off. My logic went something like this:

“If we’re already miscommunicating with one another this badly after only talking for a week over skype, then we don’t need to be friends.”

And so I stopped following him on twitter and removed him from my skype. He followed up the next day with a comment as to how he was unsurprised that I was cutting and running, and so I followed up with a final comment on how I felt it wasn’t worth the effort to sort something out with someone I was miscommunicating with, someone who was judging me so sweepingly after having only known me for a week. I said that he didn’t do anything wrong, and that he was just being him by trying to be friendly, but that we couldn’t be friends if we were unable to communicate properly after such a short time of knowing each other. And then I let him have a last word in case he needed to call me something mean to gain some sense of closure over the whole thing.

And his final follow up was just resignation, that he’d tried but that I didn’t give him a chance and wasn’t interested in giving him a chance, so what was the point? It came across sincerely sad, and that’s how it ended.


 

In all honesty, I’m not very happy with how that turned out. Contrary to what he thought, I figured him to be a very nice fellow and I was looking forward to getting to know him better! But the way that I look at it, he just became… off putting. When I have a conversation with folks, often times I just kinda drop out of the conversation. My friends do it to me, I do it to them. It’s an understanding of internet etiquette among the people I keep for company that internet conversations can and do drop off. And then they can be picked up hours later, depending on the ebb and flow of inspiration. It’s not a bad thing. It’s not really a good thing either, but that’s neither here nor there.

I’d be conversing with this fellow, and then I’d drop off cuz my mind is elsewhere, or I’m reading something, or I’m doing something, and so that was that. But I get the sense now that this was not something he really expected, nor was that the way that he managed his conversations. I guess his was a more traditional way of communicating that didn’t really fit in with mine. So the conversation would drop off, and eventually he would come along with a comment such as “o-o” or “Um.. ok.” Dragging me uncomfortably back to the conversation while I was doing something else, I guess.

And all I wanted him to do was relax. And all he wanted me to do was relax. We both got the impression from one another that we were hyperventilating at each other across the interwebs. But I’ve already talked a lot about that.


 

I guess the thing that has really been sticking in my mind about this guy is how he felt I wasn’t interested in what he was doing, that I wasn’t expressing interest despite all his attempts to express interest in me and what I was doing, and also his attempts to be friendly in general that I was apparently rejecting for no reason.

See, this sounds very familiar to a feeling that I feel often enough to be disturbed by it. I’ll meet someone, and I’ll develop a “friend crush” on them. Don’t know what a friend crush is? It’s when you meet someone, develop a strong affinity for them rather quickly, and then you want to be friends really badly.

I’ve had friend crushes and real crushes on a fair few number of people. Even folks on twitter! And those folks probably already know, even if I haven’t told them. Some of the more astute ones have had to confront me about it, actually, particularly when I start leveling bitterness at them. What sort of bitterness? That bitterness that my attempts at friendship aren’t being returned with the same level of passion and feeling that they’re being given. Sound familiar?

Man, that’s a topic for a whole ‘nother blog, to be honest. But needless to say that I’ve had friend crushes on a lot of people that didn’t turn out, and that I ended up being really bitter and upset about it afterwards for a very long time. And although I was aware of it, I stewed in my bitterness. My internal monologues complained about how unfair it was that such a neat fellow like me could be rejected for friendship by such neat fellows as them, and I glared in jealousy at their friendly interactions with their circle of friends that I apparently wasn’t privileged to be a part of.

And, here I am, facing the same sort of situation, only this time I’m on the other end. This time, I’m the “villain.” I’m the one doing the rejecting, and I’m the one basically not giving a neat fellow a chance to be my friend, despite his best efforts. In all the hindsight, in all the readings of our conversations, this bit of bitterness does appear to play a role.


So, what’s the point of all this rambling? I guess the point is that we all go through this cycle with loads and loads of people in our lives. We meet neat people who don’t want to be friends with us quite as much as we want to be friends with them. We all deal with folks who aren’t quite as into the friendship as we are, who aren’t following up on our energy and who aren’t returning our excitement.

At the same time that I am resenting Person A for ignoring my attempts to be a good friend, Person A is complaining about this same behavior from Person B, and meanwhile Person C is resenting me for neglecting their attempts at friendliness. And in the end, all seem to completely ignore their own complicity in treating in the exact same way that they do not want to be treated.

No one likes being ignored by someone they like. This is a fact. Everyone will ignore someone who likes them in their life for one reason or another. This is also a fact. People off put one another, some folks fall out. Some folks move on to other scenes and leave old scenes behind. We all do this, and we all do it equally on all sides of the spectrum. We ignore, and we off put. We neglect, and we resent.

So if everyone does this, and if we’re all equally guilty of neglecting and resenting and bittering at everyone else at one point or another in our lives… well then what?

Here, I find some acceptance, and I define a new truth… I guess? I suppose that the truth to remember is that we all have folks that we don’t get along with, and we all have folks that we do get along with! It’s like that like-level concept I talked about some blog posts back. Sometimes, you like someone at level 10 and they will only like you back at level 5, and you just have to live and let live.

Except in this case it’s more like “Some folks don’t get on with you. Let it go. And you won’t get on with some others. Tell them directly when you know.”

I dunno. This is all rambling nonsense at this point, mostly therapeutic for me, but nonsense all the same. Writing is an excellent venting form for me, and having all this space to write in does me really good in the end. I appreciate it. Hopefully some of you guys will too.

 

Tech-tech Musings: sed

Ah, sed. What a wonderfully confusing command you are~

This was one of those commands that honestly intimidated me a bit when I was in class for my Linux certification. It’s odd, has weird syntax, can be confusing to get working, and is often unneeded for basic command line tasks that most people will be doing.

But I’ve found ‘sed’ to be a really, really handy command and I sincerely enjoy using it! I’d like to explain why in this little post and hopefully give some folks an idea on how they can use it.

First, let’s take a look at sed’s man page.

NAME
sed – stream editor for filtering and transforming text

SYNOPSIS
sed [OPTION]… {script-only-if-no-other-script} [input-file]…

DESCRIPTION
Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipe‐
line). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the
input(s), and is consequently more efficient. But it is sed’s ability to filter text in a pipeline which particularly distinguishes it
from other types of editors.

So, from this we understand that ‘sed’ means “stream editor.” It’s designed to perform text edits of a “stream” on the Linux command line. “Well, that’s cool and all Jesse, but what even is a stream?” What even, indeed. Well, to understand that, you need to understand that the Linux command line has these three magical things called STDIN, STDOUT, and STDERR.

STDIN is easy enough to understand. This is “standard input,” the stuff you type into the command line to make things go, like commands or responses to a prompt/program. You type it in, press enter, and it’s input into the computer.

STDOUT is also fairly straightfoward. This is “standard output.” Any time that you run a command, you will generally get some form of text output to your command line, be it general information (like a directory listing), file contents, system information, or some sort of output that you designed your program to put out.

STDERR is a little weirder. As you might have gathered by the trend so far, this is “standard error.” This is the text that will be printed to your terminal if there was some sort of error with your command, program, or what not. The reason that STDOUT and STDERR are separated is so that if you are doing something with the text of STDOUT in your command, and then you have a problem with your command or your manipulation of STDOUT, the error stuff will remain unmanipulated in its own separate stream.

Aha! So there’s the concept now! STDOUT and STDERR are essentially text streams to the command line. It’s the text that flows to your screen, which can then be manipulated to whatever end in your command that you’re running, depending on what you’re trying to do. They can be redirected as input to files, modified by other commands, or just left to sit on the screen. I’m going to show you some examples below.

 

Example of STDOUT:

“free -m” — this command will tell me the memory usage of my system (in megabytes) at the given moment I run it. My typing it in and pressing enter is STDIN. All the goo that comes out is STDOUT.

[prannon@prannondesk ~]$ free -m
total used free shared buffers cached
Mem: 9991 9843 147 97 165 595
-/+ buffers/cache: 9082 908
Swap: 5063 14 5049

Example of STDERR:

Now I’m going to run this command with some invalid options. These options weren’t built into the program, so whatever I get out of it should throw some errors. The goo that comes out of this is STDERR.

[prannon@prannondesk ~]$ free -sdf
free: seconds argument `df’ failed

“Well, that’s all nice Jesse, I guess I get what text streams are now. But weren’t we going to talk about sed or something?”

Yeahsure. Let’s talk about that! So, one of the things that we can do on the Linux command line is take the STDOUT of one command and make it the STDIN of another command. We do this via a process called “piping.”

“PIPING???”

Yes, piping. This thing… >>   |

It’s that weird character that’s in that key just above your enter key on your keyboard. No one ever really uses it for anything, but we use this all the time to link different commands together. The pipe basically takes the STDOUT of a previous command and turns it into the STDIN of another command. Let me show you~

I’m going to use the “free -m” command from above, and pipe it to some weirdness that’ll take the word “free” in the output, and then run that as a command in and of itself. Watch…

[prannon@prannondesk ~]$ free -m
total used free shared buffers cached
Mem: 9991 9807 183 117 158 631
-/+ buffers/cache: 9017 974
Swap: 5063 18 5045
[prannon@prannondesk ~]$ free -m | grep -o free
free
[prannon@prannondesk ~]$ free -m | grep -o free | bash
total used free shared buffers cached
Mem: 10231128 10043332 187796 120588 162648 646428
-/+ buffers/cache: 9234256 996872
Swap: 5185532 18464 5167068

“Woah! What did you just do????”

So, I took free -m. The STDOUT of that command includes the word “free,” in it, which just so happens to be the same name as the command I just ran. I piped the STDOUT to the STDIN of another command that grabbed just that one word. The STDOUT of that command is “free.” I then piped that one word into the STDIN of BASH, which is the shell that I’m using. BASH sees that, thinks it’s a command, and then runs it as though I had typed it in myself.

So, here we have the idea that we can manipulate the text streams that come out of commands that we run. Now I think we can actually talk about what ‘sed’ does and what it’s useful for!

Let’s say that I want to take the STDOUT of one command, and then change it up? I can use sed to make changes to the formatting, or the spelling, or the words used in STDOUT. All I have to do is pipe the STDOUT stream into the sed command, specify my options and parameters, and then the STDOUT of sed will be whatever I wanted it to be.

Sed’s syntax works something like this…

sed ‘s/something in the original stream/whatever you want that changed to/g’

First you call to the command, and then you can have single or double quotations. Then “s,” then some dividers of some kind (commonly a forward slash or “/.” You have the option of using different dividers if it makes it easier for you to read), and then “g.” The “g” part of optional. If you don’t specify the “g,” then your change will only take place in the first instance in the stream. If you do specify the “g,” it basically makes your specified change “global.”

Neat! You can feed a stream into sed, specify what you want changed within sed, and then get what you wanted as the STDOUT of sed. Let’s see this in practice…

I’m going to use “free -m” again, but I’m going to change the first instance of the letter ‘e’ to the letter ‘a.’ I’m going to use % as my divider, since it’s easier on the eyes than /.

[prannon@prannondesk ~]$ free -m | sed ‘s%e%a%’
total usad free shared buffers cached
Mam: 9991 9830 161 143 135 689
-/+ buffars/cache: 9004 986
Swap: 5063 20 5043

Now I’m going to change EVERY instance of the letter ‘e’ to the letter ‘a.’

[prannon@prannondesk ~]$ free -m | sed ‘s%e%a%g
total usad fraa sharad buffars cachad
Mam: 9991 9834 156 143 135 689
-/+ buffars/cacha: 9009 981
Swap: 5063 20 5043

Now I’m going to be completely immature and change every instance of the word “buffers” to “poopie.”

[prannon@prannondesk ~]$ free -m | sed ‘s%buffers%poopie%g’
total used free shared poopie cached
Mem: 9991 9849 141 143 134 689
-/+ poopie/cache: 9025 966
Swap: 5063 20 5043

“Ok, that’s neat, and I guess I understand what the command is doing, but how is this even useful?”

Well… I like to think that half of Linux administration is just in modifying configuration files. There is an awful lot of configuration file modifications to be made when you’re using a Linux based OS. Sed finds its greatest utility when you’re scripting, trying to find an easy way to quickly script some changes to a configuration file, or a series of configuration files that all need the same change.

In my case, I have a program on my computer that I use to screencast my desktop. That program has a configuration file, and when I turn it on I use a script to do so. Depending on how I run the program, I may need to have some changes made to my configuration file, either to enable sound or to disable it. I don’t want to have to access my file and make changes to it every single time I run this script, so I use sed to make the changes for me.

#If the user selects to have sound, rewrite the ffserver.conf file, stop the firewall, and start ffmpeg with the appropriate options.

sed -ri ‘s/#+NoAudio/NoAudio/’ /etc/ffserver.conf
sed -ri ‘s/NoAudio/#NoAudio/’ /etc/ffserver.conf
sed -ri ‘s/#+AudioBitRate/AudioBitRate/’ /etc/ffserver.conf
sed -ri ‘s/#+AudioChannels/AudioChannels/’ /etc/ffserver.conf
sed -ri ‘s/#+AudioSampleRate/AudioSampleRate/’ /etc/ffserver.conf
echo -e “\n\nStarting Stream w/ Sound!\n\n”
systemctl stop firewalld.service
nohup ffserver &
nohup ffmpeg -video_size 1920×1080 -framerate 20 -f x11grab -i :0.0 -f alsa -ac 2 -i pulse http://localhost:9537/screencast.ffm &

In this case, I’ve added the ‘r’ and ‘i’ options to sed, which tell it to understand regular expressions (a topic for another time) and to write whatever changes it makes to the stream to the file I’ve given as its input (that file at the very end). The # you see in this example is basically a marker. It tells the program that whatever follows is to be ignored. We call this a “comment,” since most of the time the # precedes some sort of comment made on the code of the program.

A very simple edit, but it plays a very big role in my script, and it wouldn’t be easily possible without the ‘sed’ command. 🙂 Hopefully this’ll give you aspiring admins out there some ideas.

Linux Tech-tech for the Tech-tech Minded!

Hey! So, I’ve neglected this blog for long enough, and I think I’ve finally found something that I can talk about for now. Lots and lots and lots and lots of things have gone on in my life over the past few months that I’d consider blog-worthy, but I haven’t really been talking about it in a public forum largely to keep things private.

What I do wish to say is that I’m doing extraordinarily well in my new job. I feel like I’m surpassing even my own expectations with how well I’m doing, and to some degree I’m kinda afraid that I’m going to burn brightly and peak out very quickly, while everyone’s expectations of me keep climbing. But oh well, I’ve passed my own test and I feel like I’m acclimating to the new job very well.

The other thing that I want to mention on a personal note is that I’m intending to move out of my current apartment here soon. My new job affords me the ability to live alone and keep paying the bills. Aside from things that I will not discuss here, I’m mostly looking for a change of environment and a place to truly call my own. I’m actually really excited about it to a certain degree! Right now most of my things are stuffed into a teeny tiny bedroom at the current spot, and with a whole apartment to myself (even if it’s a small studio) I can spread myself out just that much more. I love changing things around and stuff, so it has me excited.

Ah, but now I will move on to other things. Past few days I’ve been writing on facebook and twitter about Linux technology things. I don’t really have a purpose in doing it. Just want to share knowledge in little snippets. I know someone out there is enjoying the reads. So, below I’ll provide the first few posts below and I’ll probably update this blog with additional items moving forward. Enjoy!


OPENVPN: 

OpenVPN is really neat. It’s probably one of the best documented technologies I’ve played with so far. They’ve got a very well fleshed out getting started page here: https://openvpn.net/in…/open-source/documentation/howto.html

What makes OpenVPN so neat is primarily this. You can set up a network of sorts that’s completely isolated from other networks. The server will be your router, and it’ll hand out virtual IPs using “tunnels” to all of the clients that you connect to it.

Where is this useful? Well, consider that you have an ISP with dynamic IPs. Your public IP is always changing, but you want to be able to host things within your home LAN as though you have a static IP. So, you buy a small server in a datacenter somewhere (such as with cloudatcost.com), install openvpn and create a server, and then create a client on your home device to that openvpn server.

Even though your ISP IP is always changing, the tunnel IP that openvpn gives your device will never change, and ideally the public IP of that server you bought in the datacenter will never change either. Bam! You now have a static IP address for a “router” that will connect your home device to the web, and you’ve gotten around your dynamic IP address problem.


FFMPEG: 

I have a love-hate relationship with FFMPEG. It’s probably the coolest and most frustrating thing I’ve dabbled in so far. FFMPEG is the technology I use to power my infamous “Lizard Cam.” You know… It’s that tech-tech that makes this thing work:http://chopstick.lizard.camera/

So what is FFMPEG? FFMPEG is basically a “transcoder.” It takes video or sound input, and then recodes it into another format/size, and it can do this all over the command line. For example, it can take raw input from a web cam and recode it into flash video formats. Often, the formats are referred to as “codecs.”

That’s where things get tricky, though. There are a gazillion different codecs for both video and audio. Some are only compatible with a select few. Like, AAC audio codecs won’t be compatible with flash codecs. Only mp3 will be. And then there’s a matter of whether you have compiled your FFMPEG program with support for the codecs that you’re intending to use. And then there’s the matter of figuring out the proper syntax for the command that you’re going to use to grab and transcode your audio/video input. And THEN there’s the matter of figuring out a working configuration in your FFserver (more about this in a minute) file so that you can stream it.

It very very quickly gets complicated. The documentation isn’t any help either, because it focuses more on concepts rather than a good “getting started” or on definitions, commands, compatibilities, or working setups. Then again, the documentation looks different now than it did when I first started dabbling: https://www.ffmpeg.org/documentation

FFserver, on the other hand, is the portion of the technology that allows you to stream whatever it is that you’re capturing. I wouldn’t say that it’s any easier, but it does kinda put itself together once you figure FFMPEG out.

All in all, very very neat technology, but exceedingly frustrating and poorly documented (as of a year ago, anyway).

WORKING FFMPEG EXAMPLE: 

I’m now going to present a working FFMPEG command, along with explanations of all the little bits.

nohup ffmpeg -video_size 1920×1080 -framerate 20 -f x11grab -i :0.0 -f alsa -ac 2 -i pulse http://localhost:9537/screencast.ffm &

*nohup = This detaches the command from your current terminal session. It writes the output to a file called nohup.out, and if you close your terminal session the command will stay running.
*ffmpeg = This is the actual ffmpeg command. Everything following that is a modifier/option that changes its behavior.
*video_size = This sets the size of the frame that your video will be, in this case a full 1920×1080 pixels.
*framerate = This is fairly self explanatory. This tells FFMPEG how many frames to grab per second. The higher the framerate, the higher the quality, but the higher the bitrate and the more internet bandwidth your little stream here will require. In this case, the framerate is 20, which is about standard.
*-f x11grab = The ‘-f’ stands for format. In this first case, we’re telling FFMPEG that our video format is going to be x11grab. In Linux, any desktop GUI environment is basically generated by a program called X11, so this tells FFMPEG to grab the video input from X11 directly. A screencast, if you will.
*-i :0.0 = In X11, the computer screens are referred to as displays. The -i tells FFMPEG to get input, and :0.0 refers to the first display. :0.1 would be my second computer screen. So basically, grab input from the first computer screen (in my case, my left screen).
*-f alsa -ac 2 -i pulse = Gonna do this as a whole, since they all go together. This time, the -f (format) tells FFMPEG about what audio we’re going to capture from the system. Alsa is actually a part of the Linux core kernel, and it interfaces with other programs to give a Linux computer sound. So we’re telling it to get stuff from the sound system directly. ‘-ac 2’ tells FFMPEG to create two audio channels. Meant to be played over two speakers, basically. Finally, the ‘-i pulse’ is telling FFMPEG to capture the sound from the Pulse Audio system. It’s similar to specifying the exact screen that we want our video from, like I described above.
*http://localhost:9537/screencast.ffm = FINALLY we have our output file. All that stuff that I described above is basically specifying inputs to FFMPEG and modifying that input’s parameters. This is where all that garbage goes after FFMPEG has processed it. The screencast.ffm file is a dynamic streaming file, which is designed to be streamed live over the internet for other viewers.
*& = This little ampersand basically runs your command in the background of your terminal session, so that you can run additional commands while this one runs. It sorta goes hand in hand with the nohup portion at the beginning.

An exhausting command, but a working one! As you can probably guess, this is designed to grab my actual desktop screen and stream it to the internet. If I were actually running this, you’d be able to see it here:

https://prannon.net/screencast

I originally came up with this because I couldn’t find a good Linux screencasting option out of all the options out there (like Livestream or Ustream, etc). Ended up just making my own.


LINUX COMMAND LINE:

First, about “command lines” in general. When I think of a command line, I think of that blinky cursor thing that appears on a black screen, and you type weird voodoo into it to make things happen on a computer. I would imagine that most people probably associate the command line (or command prompt) with hackers, computer geniuses, geeks, nerds, and things like that. But mostly hackers.

http://hackertyper.com/

The command line is actually not a very difficult concept to grasp if it’s properly explained. Let’s break it down…

1) A command line is a place where you input commands to a computer.
2) The commands that you input are defined by a “shell.” I like to think of a shell as a type of programming language. It has a syntax (way of writing commands), defined commands that you’re allowed to use, logic, and so forth.
3) The most popular command line shell for Linux is called BASH.

I think the next place to take this is to think about the commands themselves, and how they work in general.

Most commands are actually little tiny programs in and of themselves. They perform very simple tasks, and their behavior can be modified within the overall command that you feed into your command line. Let me give you an example:

“ls” — This command is a program designed to list files and folders within a certain directory (folder).
“ls -a” — Here, I’m using ls and I’m adding a modifier (or option) to the command, telling it to list all. The ‘a’ here stands for “all.” Often, there are hidden files and folders within a directory and this modifier will make them visible.
“ls -a /home/jesse/foldername” — Here, I’m specifying a specific folder path that I want to have listed.

You see how I’m doing this? I start with my command, and I add modifiers to it, and it performs a task that I want it to perform. Easy to grasp conceptually, but much more complicated once you get into the more complicated commands, and you start stringing them together into complicated scripts (or ‘one-liners’ as some people call them).

High Memory Usage on Linux

So, past few weeks I’ve noticed a distinctly high RAM usage on my F19 desktop. It wasn’t showing up in top or htop as cache or buffer, so I figured that one of my applications was using all my memory up. My kernel starting swapping programs, which resulted in horribad performance.

At first I figured it was Chrome, so I tried using a hibernation plugin to hibernate my unused tabs. No worky. Tried shutting chrome and skype down to see if that would help and still no worky. Something was just desperately filling up all that memory.

So, downloaded a lovely little program that pie graphs your memory usage by process to see where it was all going.

http://www.selenic.com/smem/

And lo! For some reason, my userspace programs were taking up less than a fifth of my memory. WTF??

Time to investigate more. As I did some more research, I discovered a Linux tool called slabtop, which apparently displays the kernel’s cache memory usage. There, I found an entry called “dentry” that was taking up a full 5GB of my 10GB RAM. This is basically a directory entry cache for the filesystem, so that the kernel can reference previously accessed files more quickly. Queue another helpful article that describes how Linux uses memory!

http://linux-mm.org/Low_On_Memory

So, did some kernel tuning to make it more prone to reclaim dentry cache and make the kernel less swappy on my machine. And I found a way to flush out the dentry cache as well, which I’ll probably do periodically anyway. Final article!

http://linux-mm.org/Drop_Caches

You Didn’t have to Cut Me Off

Anyone here listen to this song before?

I’m sure most of you have. It’s a pretty famous song, and a really famous music video.

It’s on my mind tonight. My dear, dear friend and roommate Jay Cooper has had two people – that he considered close and dear friends – cut him off over the past 7 days. No warning, no real explanation, no settling of accounts or discussion of the matter. Just blink – nothing.

It’s not a pretty sight, what I’m seeing right now. It’s very sad, upsetting, stressful, and heartbreaking to see someone going through this. I imagine it like losing my dear, dear friend Chris (there are some vacation pictures of me and him together on this site), or losing Jay himself. If those people suddenly decided, out of the blue, that I was a horrible person, blocked me everywhere, refused to talk to me, and so on… well, I’d probably break down and be a huge mess for a very long time.

Not pretty.

So, to all of you out there. Don’t cut people off. Don’t stay quiet. If you have a problem with someone, talk to them about it and settle it like as a grown man does. If you need to part ways, do so as a man does, at least by telling them why you need to part. It’s a scary thing to do, but here’s what happens if you don’t…

1) You will destroy the other person. They will have to find out for themselves, or invent for themselves, why you suddenly disappeared from their lives. Anxiety, self-doubt, and depression will destroy them.

2) You will be creating a mess that other people have to deal with and clean up. It’s selfish to force other people to make up for your disappearance, when really the only people who should be involved are the two affected friends.

Don’t cut people off. You’re being selfish, and you’re avoiding your own responsibilities to the human beings in your life.

That’s all I have to say about that for now.

Jesse, why haven’t you posted to your blog yet?

Hello all! It’s been a while since I last updated this blog, eh? Well, it’s high time that I gave a little update regarding what I’ve been up to and how the server has been handling.

The last time I posted an update, it was about how I spilled water all over the server. Well, thankfully, since then there haven’t been any issues with the way things have been running, so that’s a blessing. In the intervening time, I’ve kept busy with work and with acquiring various things. Those who follow me on twitter and facebook probably have seen the various pictures of the things I’ve gotten, but I’ll go ahead and put them here too. 🙂

Sailboat!

I got a sailboat! It’s not a very big boat, only about 11 feet long. More specifically, it’s a dinghy style Super Snark boat.

“Why did you get a sailboat Jesse?”

Several reasons, really. A friend of mine last summer invited me to go sailing with him on his much larger boat, which he kept out at Lake Travis before he gave it away. I took him up on it, and it was one of the more enjoyable and relaxing things I’d done in recent times. It also reminded me of the fun times I had in Boy Scouts, earning my sailing merit badge on Snark boats.

Sadly, being able to organize sailing trips with my friend was difficult, and it grew even more difficult after he gave the boat away to a coworker, so I settled on getting myself my own boat and just going to the lake whenever I damn well pleased! It just so happened that there was an outfit in downtown Austin that rented – and sold – various models of Snark boat. They were HostGator customers for their website as well, so it turned out to be a match made in heaven as far as being a customer is concerned. And so, I bought a brand, spanking new Super Snark sailboat for just a little more than $1,100.

SailBoat on the Wall!

 

Interesting storage solution, huh? I think it adds character to my apartment’s living room. 🙂 It makes for a good, short conversation piece, and it’s hella fun to sail around!

 

Nginx!

Now for some more technical type stuff. For those unfamiliar, most servers that serve websites these days are powered by a program called “Apache.” This is the program that takes files and data on the server’s hard drive and sends it to you when you enter a URL for a website. Apache has been around for a long time and it’s pretty much the standard bearer.

Lately at my job I’ve been hearing more and more about a new server program that is growing very popular. This program is called Nginx (pronounced Engine-X).

Why is it getting popular? Well, it has some features that Apache doesn’t, basically. What are these features?

1) It’s lightweight. It doesn’t consume many server resources, which generally means you can get better performance with large volumes of traffic.

2) Proxying. Nginx is capable of serving as a reverse proxy for another server. That means, it can act as a front end for something else. In my case, I switched my front end web server from Apache to Nginx, letting Nginx handle all the basic front end requests while Apache runs in the background.

Why is this good? Well, I’ll explain more below.

3) Caching. Nginx has a built in caching function, which is enormously handy! For those who don’t know, caching basically improves server performance. How? Normally, when a user requests a web page, the server has to do a lot of work to build the web page and then send it out over the internet. A cache is basically a pre-built, saved copy of the web page. The server does less work to build the page, and all it has to do is send the copy.

Apache is capable of building a server side cache of web pages and media, but it involves a great deal of hassle and it’s not very user friendly for an administrator to set up. Not so for nginx. Nginx actually makes the process of setting up server side caching fairly simple and straightforward.

So, here is the trick then. If you combine nginx’s proxy ability with its caching ability, you can save your Apache server some effort in building web pages and you can generally get better performance than you otherwise might. My configuration is probably a bad one, but it’s neat to set it up from scratch, see it work.

***

I have more things that I could continue to write, but I’ve run out of time for now. I’ll probably write another update soon!

 

So a Thing Happened to the Server

So, and interesting thing happened just this past Friday while I was messing around playing games on my computer. I spilled water on my server.

For those who do not know, I currently use a Dell PowerEdge 1850 rack mounted server as my general purpose server for the things that I do. This server provides a platform for this website, my LizardCam, an IRC server, a Ventrilo server, and a Plex Media server that I use to enjoy my various movies and videos and things. I keep it on a desk right next to my computer. Since the actual machine is really flat, this works out well and I can just put stuff on top of it like it were a part of the desk.

Now before you think that I’m putting water on my server, no, I’m not. I’m not quite that stupid. I’m still pretty stupid though. I was keeping a big jug of water right neeext to it. And I was turning my chair and it tipped over right onto the machine. It was pretty bad and very stupid.

Now, the first instinct at the time was to turn it off. Immediately. I did a hard stop on the box and opened it up and checked for moisture and water. For the most part, it looked like it was fairly localized around the left hard drive and on the front plane. I took a few pieces of paper towel to it to try and dry it out, and removed some pieces to check for any moisture here and there. I gave it a little while to dry off, and then tried to restart it. All went well for a few minutes while the machine went through it’s various start up checks, but when it tried to spool up the disk drive for the start up check, it turned itself off.

Now, the neat thing about these Dell machines is that they will turn themselves back on after a power outage or after any event that causes them to switch off. It tried to turn itself back on. Once, then twice, and then it stayed off. That was really weird. And any time that I tried to turn it back on after that, it switched itself off again. So, at this point, I was really scared that the machine was a lost cause. I tried everything. Tried hooking up a monitor to it so that I could see any errors and got nothing. Tried to switch which power supply I was using and also got nothing. It would power up the fans for a short bit, and then switch itself off again in very short order.

So yeah, I was pretty certain that the machine was toast at that point, and it felt awful. But, I sought counsel from my various friends who are familiar with Dell products and hard ware in general. The consensus was that “Yeah, it might be toast, but you need to let it air out and dry for a few days before you try starting it up again.” So, that’s what I did, despite how impatient I was to try and restart the damned thing. The whole incident happened on a Friday, and I let it air out until Monday morning when I tried restarting again.

And here’s the really cool part. It started up and ran as though nothing had happened. I was fully expecting that it would not work, but it did! I was astonished! Very impressed with the hardware!

Oh yeah, and I’m going to try and be less stupid with this machine now. I’ve put it up on some stilts so that it sits above the desk instead of on it. That way, if there IIIS spillage, it’ll just go under the machine instead of on top of it.

So yeah, that happened.