Your browser (Internet Explorer 6) is out of date. It has known security flaws and may not display all features of this and other websites. Learn how to update your browser.
X

MyProcurement: All government tenders in one Excel file

MyProcurement

Happy birthday Malaysia!! Just how awesome is our country, that we celebrate an Independence Day AND a Malaysia Day, not to mention 2 New years day, (or 3 if you count Awal Muharram).

So on that note, I decided to use my IT skills for the good of the country.

To be honest, my IT skills have never been up to par, my day job is more managing/planning/documenting than actual execution of ‘real’ IT work. But it was good for me to dust of the ol’ programming fingers and learn Python to grab some publicly available information and make it more accessible to the less IT centric members of society.

Since I had limited time, and sub-par skills, I decided to set my sights low, and aim to extract all the data from the Malaysian MyProcurement portal, which houses all the results of government tenders (and even direct negotiations) in one single website for easy access. The issue I had with the portal though, was that it only displayed 10 records at a time–from it’s 10,000+ record archive, so there was no way to develop insights into the data from the portal directly, you had to extract it out, but the portal provider did not provide a raw data dump to do this.

So I wrote a simple Python script to extract all the data, and prettified the data in Excel offline. The result is a rather mixed one.

I was happy that I could at least see which Ministeries or Government departments gave out the most contracts, and what the values of those contracts were. All in all, the excel spreadsheet has more than 10,000 tenders with a cumulative value of RM35 billion worth of contracts going back to 2009. The data allowed me to figure out which Ministry gave out the most contracts, the contracts with the highest and lowest value (including one for Rm0.00, and one for just Rm96.00). All in all it was quite informative.

Results_by_ministry

More…

Is Malaysia’s Broadband slow–no it isn’t.

Broadband_speed_klang_malaysiaRecently KiniBiz did a piece on Malaysian broadband speeds, and once again the hoopla about how Malaysian broadband speeds are slow arose. Kinibiz quoted an article from Asean DNA which stated that the average broadband speed in Malaysia was just 5.5 Mbps, while Thailand, Vietnam and Singapore had speeds that were double that (or more!)

The report however was inaccurate, and I think there’s a need to address the hoopla, because this happens often. There was a report couple months back that said Cambodia had faster speeds than Malaysia, and I wrote a post addressing that. This time I think, we have to really go into the data and find out what exactly is going on.

So let’s start at the source of this data.

The data was built from billions of download test conducted by users throughout the world on speedtest.net (a website that allows users to test the speed of their internet connection). This dataset is HUGE!, one of the biggest I’ve seen and definitely the biggest I’ve had the pleasure to play around with. Just one file in the set had more than 33 Million rows and weighed in at more than 3.5GB.It took me some time and lots of googling just to figure out how to deal with a csv file this large. Fortunately, there’s LogParser, but we’ll skip that tutorial for now and focus on the juicy details of data.

The number reported by Asean DNA is wrong. The average internet speed in Malaysia isn’t 5.5Mbps, it’s more like 7.5Mbps.

5.5 Mbps was obtained by averaging the speed across the regions of Malaysia (Kl, Alor Setar, Klang..etc) rather than by averaging the speed across all the test conducted by Malaysian users. In short, Asean DNA placed equal emphasis on Kuala Terengganu and Kuala Lumpur, although Kuala Lumpur had 50 times more test conducted. It would be like calculating GDP per state, rather than GDP per capita. The real per capita download speed in Malaysia is 7.5Mbps, rather than 5.5Mbps (if you limit yourself to just data from 2014).

Here’s the breakdown. You can download the file from netindex.com or just use an extract I created with just the Malaysian data–it took some time to do this so leave a Thank you in the comments if you downloaded the data.

Average-speed-internet-Malaysia

More…

A Techie’s view on the Law

2283584007_f199332890_zAre some laws worth following–in other words, are some Laws so idiotic that they should be ignored completely?

That sounds anathema, because we have a romanticized definition of the law, we define the Law as a broad general agreement a society undertakes, and the law keeps society from tearing itself apart. In other words, the law is so sacred because without it–we descend into anarchy, so ignoring the law is akin to promoting anarchy.

But I’m not speaking of “The Law”, I’m speaking of “A law”, specifically an Act of Parliament. “The Law” refers to a vast conglomeration of many things, including constitutions (state and Federal), statutes, precedence of case law and Acts of Parliament. I’m not sure what a statute is–but I roughly know what an Act of Parliament is, and it surely isn’t a broad general agreement that society depends on to stave off Anarchy–rather an act of Parliament is a law brought into effect by Parliament–nothing more nothing less.

To my techie mind, that means that 222 Members of the Malaysian Parliament got together to enact a piece of legislation. Romantically we think this is the people’s will–the Rakyat voted these people into power and they now wield this power to enact laws that will protect the Rakyat. A glorious cycle of virtuosity that only democracy can deliver. That’s wishful thinking, realistically it’s a law brought into effect by 222 voting members of Parliament whose collective IQ would probably not exceed that of the Zoo.

So when these 222 MPs ge t together and enact legislation to regulate technology–I get a bit uncomfortable. Not only do most of them not have engineering qualifications, half of them don’t even have a website. Having these MPs enact legislation that will regulate a field they’re clueless about, is akin to getting open heart surgery from a car mechanic.

On a side note, a techie like me has a hard time understanding why we have 222 seats in Parliament. It would seem, that in a first past the poll system, you’d want to have ODD number of seats, to avoid the situation where 111 members belong to Barisan, and the other 111 belong to Pakatan (what happens then?). That’s just ONE of the many things an engineer would quickly realize is wrong with the entire system–and that’s why we only have 3 engineers in Parliament (at least according to the Sinar Project). More…

Nearlyfreespeech the hosting provider that takes security seriously

NearlyFreeSpeech.NET Web HostingYou all know how much I love nearlyfreespeech, it’s one of the best hosting providers out there. Here’s one more reason, recently they alerted me to a suspicious number of login attempts to my wordpress site, which usually means someone was trying to hack it.

If you remember the post I did about the RHB bank scam, it’s quite common for hackers to inject pages onto a wordpress site to help them carry out banking scams. This was probably something similar.

Fortunately, the guys over at nearlyfreespeech were not just kind enough to log the attempts and alert me, but even automatically disabling the login page of the site to prevent something similar happening. Good on them!

Nearlyfreespeech is a great hosting provider and this just proves my point. Check out the email below: More…

Why the ban on Uber means more than just Taxis

Uber in MalaysiaUber, a company that connect passengers with drivers of vehicles for hire and ridesharing services–is now banned in Malaysia. The Uber service was viewed as a god-send by citizens of KL simply because the existing taxi service in the city–is Shit!

So why did the Government choose to ban a service, that was improving public transport in KL? For the small matter that it violated a couple of laws–no big deal really. Laws are made to be broken aren’t they?

There’s no need to go into the details of whether Uber is legal or illegal, that’s an irrelevant point. Even though the cars Uber offered were far safer and more comfortable than any other Taxi on the roads in KL–it probably couldn’t get the necessary licenses if it tried. In other words–it couldn’t legalize itself. But that’s a mere side-note to this whole debacle, the real problem is how Malaysia handles disruption.

Uber is a disruptive service that was on the verge of changing the way Malaysians view public transport–but some people in high places may not have liked that. To me as a techie I can’t understand the reason for any of these laws–in fact at its core ‘The Law’ is a piece of technology. But the people drafting the law, MPs in Parliment aren’t even well versed in things like the internet, let alone these disruptive technologies which requires new laws to regulate.

If we are to be a developed nation, we need to embrace disruptive technologies, because that’s how we innovate, and at some point all innovation requires someone to break the law, because the law can’t foresee disruptive technology–that’s almost by definition. We can’t realistically expect the politicians to keep up with technology, let alone draft legislation to regulate them. And every new piece of legislature put out by politicians very quickly reaches the boundaries of it’s effectiveness the moment new technology becomes available.

Consider the following:
More…

What do ISIS and Genghis Khan have in common?

Image courtesy of www.breitbart.com
Image courtesy of www.breitbart.com

I thought I’d take a break from writing about Tech this week, to focus on where the worlds attention should be–Iraq and Syria, and the existential threat that is besieging the middle-east, a threat we’ve come to call The Islamic State of Iraq and Syria (ISIS).

ISIS may have started of an off-shoot of Al-Qaeda, but they’ve evolved to be much more, they’re so far removed from the Al-Qaeda of Osama Bin Laden they’re starting to look like an reincarnation of an much for terrifying ancient enemy. An enemy that 700 years ago threatened the very existence of Islam in the Middle East more than Israel ever could today. ISIS might be just be the next–Genghis Khan.

You might scoff and laugh, but there are many things strategically and tactically that make ISIS look like a direct descendant of Genghis Khan, and there were many contextual similarities between what’s happening now in the Middle East, and what was happening 700 years ago when the Mongol Horde descended upon it.

The Mongols are the exception to all of history, for example they’re the only people to have invaded Russia in the Winter–and WON!, and they invaded Afghanistan on horseback–and WON!

Historians will glorify them, but we know better. At its peak the Mongol armies controlled more land than anyone before them , but they won their battles in vile, vicious and cunning ways and what they did to their captives is unspeakable. Truthfully the Mongols (and specifically Genghis Khan) were Assholes of the highest order–but they weren’t the only Assholes in history. Alexander, Napoleon, Attila–these were all assholes, it’s just that the Mongols won more wars giving them more opportunities demonstrate their asshole-ness.

Of course the Mongols weren’t just assholes, they were phenomenally great Military Tacticians and Strategist. Almost everything they did (apart from binge drinking) was for the sole purpose of winning wars and battles–even their rapes and terrorizing served to aid them in victory over the enemy (and we’ll soon see how).

But when you dissect as to why the Mongols were so successful–you’ll soon realize it’s the same reason ISIS is so successful, and the similarities between ISIS and the Mongols are too shocking to ignore. More…

Is illegal downloading considered stealing?

copying_is_not_piracy

There’s a general perception that illegal downloads of movies, music or books is akin to stealing the works of the artist, singers and authors. But downloads are more akin to trespassing than it is stealing–they’re nothing like stealing and if you take some time off to think about it, you’d figure this out too.

When you steal something, you’re denying the victim something,if you steal my car, you’ve taken something from me that I can no longer use. Stealing is a zero sum game, where the perpetrator gets something, and victim loses something. Obviously stealing is bad, but downloading isn’t stealing.

When you illegally download music, you’re not denying anyone else something physical. The music that you downloaded is still on the server you sourced it from, all you did was make a copy. Illegal downloading, isn’t zero sum, you profited (maybe),but no one lost anything of value. And so the rules that we devise for digital information (which can be replicated freely) shouldn’t be the same rules we apply for physical items like cars, and gold coins (which can’t be replicated freely).

Of course, the argument is that illegal downloads deny the content creators income they rightfully deserve, but that’s only true in a handful of cases. If someone refused to sell you their content regardless the price you’d be willing to pay–does it then make it morally wrong for you to download the material? You cannot consider it a loss for the content creator if they never intended to sell the item to you in the first place.You cannot deny income to someone who refuses to take your money, so therefore an ‘illegal’ download isn’t stealing at all.

And it doesn’t stop there. What happens if the content creators are just plain jerks–would it be morally wrong to copy their digital data–and would that be considered stealing? More…

CheDet on Censorship

Tun Dr. MTun Dr. Mahathir now says he’s change his mind about internet censorship. To quote him “Not knowing the power of the Internet, I promised that we (speaking as the Prime Minister of Malaysia) would not censor it. But today I have changed my mind.”

Of course, everyone has a right to change their mind–but in this case Tun went from being absolutely spot-on (the internet doesn’t need censorship) to dead wrong.

The first thing Tun says, is that Internet is already censored, because his blogs were blocked by various internet companies.

Admittedly, Facebook and Google have too much power over the content we access on the internet, if something doesn’t exist on your Facebook feed or Google search, it might as well not exist at all. And granting that much power to a single organization (whether private enterprise or Government) is detrimental to our ability to access information.

But, Tun should have admitted that being on the other side of that stick, he finally saw that the power to censor information is too much power to be concentrated in the hands of a few Facebook employees (or government servants), and hence he was right to make the internet in the Malaysia censorship free, because even the Government shouldn’t be given that much authority.

Sadly, he didn’t admit that, instead he resorted to a “They can do it, so can we” attitude. Haih!

Internet censorship is bad, Tun himself admits it, when he complains about Facebook censoring his blog post (specifically post about Jews and Israel). This isn’t the first time either, I wrote about Facebook censoring chedet.cc more than year ago–I guess the guys over at Facebook took offense that Tun doesn’t distinguish between Jew and Israeli. When you write a blog post criticizing Israeli Foreign Policy it’s probably wise to NOT to title that post “The Jews”!

However, all that aside, there is HUGE difference between an individual company like Facebook censoring content on their servers, and a government censoring content on other people’s servers.

Facebook and Google are so powerful on the internet, that maybe they’re responsibility should extend beyond the corporate realm and into the sphere of public interest. After all, people in the US call 911 whenever Facebook goes down, it’s that important to them, but that’s a separate discussion, individual companies can do what they want so long as that is inline with the law. Governments on the other hand, need to abide by the principles of Freedom of speech, simply because they have more power.

To illustrate the difference, let’s take Malaysiakini, which has been a thorn in the side of the Government for a long time. If Malaysiakini’s hosting provider, decides to no longer host them for political reasons, Malaysiakini has a long list of other hosting providers that would gladly provide them services for the right price. If Facebook blocks Malaysiakini, there’s still twitter or Google+ (not as powerful of course, but good alternatives nonetheless). On the other hand, if the Government decides to block Malaysiakini in Malaysia, suddenly there is no alternative, what can Malaysiakini do? More importantly, what could Malaysians do to access that information–nothing much! (actually quite a lot, but let’s assume for now)

That’s the other side of the equation–when Facebook censors chedet.cc, they’re not just censoring one man’s blog, rather they’re censoring 13 Million Malaysian Facebook users from reading  the thoughts of a former Prime Minister. Censoring isn’t just about preventing people from speaking, it stops others from listening, and who is the Government to say what I can (or cannot) listen to. I will decide that on my own, Thank you very much!

Do you want to live in a world, where the government controls what Ideas you can access? And Who in Government would you trust with such power–to decide which ideas are acceptable? Nik Aziz, Chua Soi Lek, Najib Razak, Lim Kit Siang, Anwar Ibrahim, or Bung Mokhtar?–who among these fine gentlemen would you pick to have complete say over what you can and cannot see on the internet? And if you can’t name one person whom you would trust 100% for control of the internet, then the entire idea of censoring is moot.

And it’s not as though we don’t already censor the internet.

Rewind less than 6 months back, and we see that mentioning a certain Vegetable got a BBC website blocked. In the run-up to the 2013 elections pages linked to opposition parties were blocked as well–and here we see the REAL reason for government censorship. It isn’t about keeping the peace, or preventing civil uprisings–it’s more about protecting individual political personalities from attack. Censorship has always been political in this country, and we no reason to believe that will ever change.

The very first time Malaysia censored the internet (officially at least), was shortly after the 2008 General Elections, and that was a directive from MCMC to block MalaysiaToday by Raja Petra Kamaruddin. Once again, this wasn’t about blocking pornography or LGBT, it was purely about blocking political news. So any mention of pornography or LGBT as Tun does, is obviously a straw man argument.

Of course this is just a re-hash of my thoughts on censorship, but the next time you see someone who agrees with the internet censorship get them to explain to you why we need it, and ask them for the data.

Photo from https://www.flickr.com/photos/ajimns/3110643778