MyProcurement: All government tenders in one Excel file

MyProcurement

I’ve updated this post on 31-Mar-2015, to incorporate the latest changes, and to provide more up to data info on the procurement database. Left everything else in tact.

Happy birthday Malaysia!! Just how awesome is our country, that we celebrate an Independence Day AND a Malaysia Day, not to mention 2 New years day, (or 3 if you count Awal Muharram).

So on that note, I decided to use my IT skills for the good of the country.

To be honest, my IT skills have never been up to par, my day job is more managing/planning/documenting than actual execution of ‘real’ IT work. But it was good for me to dust of the ol’ programming fingers and learn Python to grab some publicly available information and make it more accessible to the less IT centric members of society.

Since I had limited time, and sub-par skills, I decided to set my sights low, and aim to extract all the data from the Malaysian MyProcurement portal, which houses all the results of government tenders (and even direct negotiations) in one single website for easy access. The issue I had with the portal though, was that it only displayed 10 records at a time–from it’s 10,000+ record archive, so there was no way to develop insights into the data from the portal directly, you had to extract it out, but the portal provider did not provide a raw data dump to do this.

So I wrote a simple Python script to extract all the data, and prettified the data in Excel offline. The result is a rather mixed one.

I was happy that I could at least see which Ministeries or Government departments gave out the most contracts, and what the values of those contracts were. All in all, the excel spreadsheet has more than 10,000 tenders with a cumulative value of RM35 billion worth of contracts going back to 2009. The data allowed me to figure out which Ministry gave out the most contracts, the contracts with the highest and lowest value (including one for Rm0.00, and one for just Rm96.00). All in all it was quite informative.

Results_by_ministry

Continue reading

Is Malaysia’s Broadband slow–no it isn’t.

Broadband_speed_klang_malaysiaRecently KiniBiz did a piece on Malaysian broadband speeds, and once again the hoopla about how Malaysian broadband speeds are slow arose. Kinibiz quoted an article from Asean DNA which stated that the average broadband speed in Malaysia was just 5.5 Mbps, while Thailand, Vietnam and Singapore had speeds that were double that (or more!)

The report however was inaccurate, and I think there’s a need to address the hoopla, because this happens often. There was a report couple months back that said Cambodia had faster speeds than Malaysia, and I wrote a post addressing that. This time I think, we have to really go into the data and find out what exactly is going on.

So let’s start at the source of this data.

The data was built from billions of download test conducted by users throughout the world on speedtest.net (a website that allows users to test the speed of their internet connection). This dataset is HUGE!, one of the biggest I’ve seen and definitely the biggest I’ve had the pleasure to play around with. Just one file in the set had more than 33 Million rows and weighed in at more than 3.5GB.It took me some time and lots of googling just to figure out how to deal with a csv file this large. Fortunately, there’s LogParser, but we’ll skip that tutorial for now and focus on the juicy details of data.

The number reported by Asean DNA is wrong. The average internet speed in Malaysia isn’t 5.5Mbps, it’s more like 7.5Mbps.

5.5 Mbps was obtained by averaging the speed across the regions of Malaysia (Kl, Alor Setar, Klang..etc) rather than by averaging the speed across all the test conducted by Malaysian users. In short, Asean DNA placed equal emphasis on Kuala Terengganu and Kuala Lumpur, although Kuala Lumpur had 50 times more test conducted. It would be like calculating GDP per state, rather than GDP per capita. The real per capita download speed in Malaysia is 7.5Mbps, rather than 5.5Mbps (if you limit yourself to just data from 2014).

Here’s the breakdown. You can download the file from netindex.com or just use an extract I created with just the Malaysian data–it took some time to do this so leave a Thank you in the comments if you downloaded the data.

Average-speed-internet-Malaysia

Continue reading

A Techie’s view on the Law

2283584007_f199332890_zAre some laws worth following–in other words, are some Laws so idiotic that they should be ignored completely?

That sounds anathema, because we have a romanticized definition of the law, we define the Law as a broad general agreement a society undertakes, and the law keeps society from tearing itself apart. In other words, the law is so sacred because without it–we descend into anarchy, so ignoring the law is akin to promoting anarchy.

But I’m not speaking of “The Law”, I’m speaking of “A law”, specifically an Act of Parliament. “The Law” refers to a vast conglomeration of many things, including constitutions (state and Federal), statutes, precedence of case law and Acts of Parliament. I’m not sure what a statute is–but I roughly know what an Act of Parliament is, and it surely isn’t a broad general agreement that society depends on to stave off Anarchy–rather an act of Parliament is a law brought into effect by Parliament–nothing more nothing less.

To my techie mind, that means that 222 Members of the Malaysian Parliament got together to enact a piece of legislation. Romantically we think this is the people’s will–the Rakyat voted these people into power and they now wield this power to enact laws that will protect the Rakyat. A glorious cycle of virtuosity that only democracy can deliver. That’s wishful thinking, realistically it’s a law brought into effect by 222 voting members of Parliament whose collective IQ would probably not exceed that of the Zoo.

So when these 222 MPs ge t together and enact legislation to regulate technology–I get a bit uncomfortable. Not only do most of them not have engineering qualifications, half of them don’t even have a website. Having these MPs enact legislation that will regulate a field they’re clueless about, is akin to getting open heart surgery from a car mechanic.

On a side note, a techie like me has a hard time understanding why we have 222 seats in Parliament. It would seem, that in a first past the poll system, you’d want to have ODD number of seats, to avoid the situation where 111 members belong to Barisan, and the other 111 belong to Pakatan (what happens then?). That’s just ONE of the many things an engineer would quickly realize is wrong with the entire system–and that’s why we only have 3 engineers in Parliament (at least according to the Sinar Project). Continue reading