Cheap Pen Testing
Here's what happens when you buy cheap pentesting.
Firstly, let's establish that I am not a sales guy. I’ve been a security consultant for 6 years now and in that time, I have been part of many pre-sales calls to help add technical input into the conversation and ultimately form a scope of work to which we use to perform an engagement with a client.
Potential clients mainly come in through acknowledgement of quality research these days. However, in the earlier days, potential clients would sometimes reject our scope of work in favour of other vendors with the commentary usually being the same:
“Your technical knowledge is clearly present and would be a good fit, however another vendor fits our budget better”
Essentially, they were picking a vendor purely based on price. This is frustrating for both the sales executive and I, due to putting effort into pre-sales when I could have been researching or furthering my own learning.
In this industry, typically quotes will only differ by a few hundred pounds. Therefore, in the grand scheme of things this isn’t too much more money to go for a vendor who has shown strong technical ability in pre-sales and is a high-quality option. Based on this, I wanted to conduct an experiment.
The experiment
I was taught from my parents and learned from my own experiences that “If you buy cheap, you buy twice”. In other words, if you pay too little for anything you should expect it to break or be poor quality so will need to be bought again.
To prove this fact within this industry, I thought that I would buy some Penetration Testing.
The thing is, this is an experiment and not a real project. I don’t have a spare money tree allowing me to instruct a known cheaper vendor to do this for me, so I went down the freelance route.
There’s a website that I’ve known about for years called ‘Fiverr’. This website allows freelancers to post jobs without any kind of verification of their skillset, anyone may post a job. I simply typed ‘Penetration Testing’ into the search box:
You may quickly notice that most of the jobs shown above are crazily cheap.
£8.21 for a “Thorough penetration test of your web application” – Seven, 5-star reviews!
£12.32 for “I will penetration test your website with professional report” – Ninety-six, 4/5-star reviews!
If these freelance testers are receiving this many good reviews, buyers are clearly believing that they are getting good value for money. I’m obviously sceptical of this because can you really put that much effort into security testing for this little money? I decided that I’m going to find out.
Setting Up
Before buying from a freelancer, I needed to have something ready for the testers to assess. The first thing I did was browse Digital Ocean and find a WordPress droplet for an easy start:
I created the droplet and within minutes I had a box ready to start introducing vulnerabilities. I built the website using a free template and made it appear as though it will be used as a travel blog.
My pretext was simple. I am a woman named Emma who has had trouble with WordPress being hacked previously. Before adding any content, I would like to assure the website is secure, hence requiring an assessment.
The domain I used was ‘tryhack.me’, as I had this already bought and it was unused. I thought this may put the testers off and stop my experiment, but they didn’t even question it.
I wanted the testers to be able to find several issues, so I introduced the following:
No HTTPS
Weak SSH Password
Weak MySQL password and open on the external network
Vulnerable WordPress Plugin
Cross-Site Scripting
Ability to upload a reverse shell
The important thing for me was to create issues that were simple and easy enough to find. I’m assessing the testers ability to find any issues at all, and by making deep rooted issues I knew they just were not going to be found via a cheap assessment.
No HTTPS:
This issue was easy to introduce, I just didn’t add a certificate and left the website to run on HTTP.
Anyone with even the slightest bit of security knowledge would be able to see that the website is running via HTTP and not HTTPS. The tester should raise this as an issue due to lack of encryption.
Weak SSH Password
For this issue, I simply logged into the host with root via SSH and changed the password to ‘Password123’. I tried to go weaker than this, but the box wouldn’t allow it, so I went ‘Password123’.
If a tester tried even the simplest SSH brute force on the host, they should be able to login.
Weak MySQL Implementation
For this issue, I once again logged into the host with the root user via SSH. I then logged into MySQL.
I changed the password to a value even weaker than the SSH password, this was ‘password’.
Finally, I allowed MySQL to be accessed externally rather than just internally. This would allow the root user to be accessed via the external MySQL service, which isn’t good practice.
To find this issue, a tester could execute a simple NMAP scan and discover the MySQL service open on port 3306. Like SSH, they could run a simple credential brute force and as the password is literally ‘password’, this should be an easy find for any tester.
ThemeGrill WordPress Plugin
A few weeks before this experiment, a vulnerability was found in the ‘ThemeGrill Demo Importer’ plugin which essentially allows for the deletion of a website. I thought this would be a nice inclusion.
I couldn’t download the plugin using WordPress itself, so I had to go on ThemeGrill Github and download the vulnerable version, then upload the plugin manually. This may be harder for a tester to enumerate and find, although I deliberately left the ThemeGrill WordPress footer:
‘Copyright © 2020 SheTravels Theme: Flash by ThemeGrill. Proudly powered by WordPress’
This clearly showed that I am using ThemeGrill. I left this in the hope that they would at least mention this issue. All It would take would be a simple Google of ‘ThemeGrill vulnerabilities’ to find this exploit!
Cross-Site Scripting and Shells
This is probably the most trivial Cross-Site Scripting and Web Shell issue ever, but it has a purpose.
To include these issues, I modified a PHP file and named it s2.php. I modified the following code:
The web shell was easy. There was already no filetype validation and that would therefore allow an attacker to upload a web shell and do as they please. The modification I made wasn’t much, I just added a HTTP GET parameter that echoed out the string in the GET parameter to the page.
You will see that ‘lol’ is echoed below:
This has no input filtering and thus, Cross-Site Scripting existed:
To find this issue, I expected that the tester would conduct a directory brute force attack with a tool such as DIRBuster. This would find the s2.php page.
Once a file is uploaded using this page, I coded the script to present the HTTP GET URL, so it was easy for the tester to find this parameter. They would have everything they need to find the two basic issues.
Time to Buy
Now that I had a vulnerable platform built, I could engage some testers. To have a good comparison I wanted to select three cheap testers. I spent some time looking through all the penetration testing options available to me on Fiverr and I ended up picking three testers that had good reviews and were under $15.
One thing I noticed was that freelance penetration testers love a low-quality marketing image:
Anyway, I bought the following jobs, two for $15 and one for $5. Yes, that’s not a typo, $5.
Their job descriptions weren’t terrible. Two of them detailed they would find issues based roughly around the OWASP Top 10 and looked semi-legitimate. One of them had no description but had good reviews.
$5 Description
$15 Description
Tester Communication
Upon purchasing the tests, I gave the following job description to all testers:
Hiya, I would like you to test my website and server for any vulnerabilities. I have had trouble in the past with my WordPress being hacked, so I have setup a new WordPress. Before I add content, I want to ensure it is safe and won’t be hacked. I bought a domain to use initially while I have the testing done. The URL is http://tryhack.me
The replies quickly came in, but two of them were laughable. I clearly stated that I would like their IP Addresses for whitelisting as this is good practice with any development application. You would assume that the testers would encourage this practice.
However, the $5 tester asked the restriction to be removed:
One of the two $15-dollar testers gave a very interesting response:
Only manually done? Not private tools? What?! My favourite though:
Respect to the other $15 tester though, they did things right:
After a bit of back and forth with the other two testers, I finally managed to get the IP Address whitelisted for the $15 tester. However, they quickly came back with another IP for me to Whitelist because they needed “Deep Analysis”:
Before I whitelisted the new IP Address, I conducted a reverse DNS:
It turns out the new IP Address is a 3rd party WordPress scanner, which is not the best practice to be following on a penetration test. You should be running tools from your own IP ranges.
By scanning the website with this external tool, the tester clearly found a CSRF issue.
It’s an interesting way to describe a CSRF attack, but it’s also a false positive. If that’s not funny enough, the “I think this is the only vulnerability” might not fill you with confidence.
Basically, the tester is trying to say that they have run the WordPress scanner and now they’re out of ideas. But wait! If that still hasn’t quite got you, next up the tester asks for more money and a 5-star review!
You could say the overall experience with this tester was far from amazing. I’ve not mentioned the other testers yet. I initially whitelisted $5 testers IP Address, but then radio silence…
I had to ask if the tester had started yet and was told that they had, but they have a dynamic IP Address and thus couldn’t carry on testing. At no point was I told that they couldn’t access the host anymore, I had to ask. Anyway, I ended up having to drop the firewall for this tester which is terrible testing practice. That was all the communications I had with this tester.
Meanwhile, the second $15 tester was well away. Communication came through quickly that he had found the MySQL weak password and I was quite confident this tester was going to find everything.
Unfortunately, other than telling me I would get my report tomorrow, this was all the communication I received, which was disappointing. I was hopeful for this tester.
Overall, all three testers were not impressive with their job handling skills, but the second $15 tester handled things the best. What I would like to note, is that at no point was a scope agreed, legal documents signed or any of the normal testing formalities met.
Logs
Before we move onto the reports I wanted to show some logs which would have detailed what the testers have been doing. However, they aren’t worth showing. Long story short, they all executed HTTP scanners which was most likely CMScanner or similar.
None of them tried brute forcing the web directories and thus did not find the s2.php page.
Only the one $15 tester attempted passwords on the MySQL service, manually, which is why only he found the weak password on the MySQL service but not the SSH service.
Aftermath
I received all the reports the day after testing in the following formats:
$5 Tester – .docx file
$15 Tester – .pdf file
$15 Tester 2 – .txt file
$5 Tester
The report for the $5 tester is shown below:
As you can read, this tester reported on a lack of HTTPS which was expected. Interestingly they also reported that we are vulnerable to DDOS Slowloris attack. This is a false positive as the web application was protected by Cloudflare, so that wasn’t great.
Other than that, according to this tester, we are secure. But for good practice, they recommend that we change the WordPress login from ‘wp-admin’ to ‘admin’ to increase security. This is a bad recommendation; an attacker will find this directory in a matter of seconds. To be fair though, they do recommend using Two Factor Authentication and a WordPress security plugin which is good practice.
Overall, there is a bit of good information, but so much has been missed and they have included a false positive. By telling someone that you are secure and “Website don’t have any other vulnerabilities” is just a false sense of security which is obviously very bad and may leave the client vulnerable.
$15 Tester
The report for the first $15 tester was a PDF document created from a PowerPoint presentation. It was quite hard to read and was misleading.
First page, “Your WordPress website is vulnerable to attack!”:
Page two seems to come a bit early, conclusion?
Okay, so the website is vulnerable to attack. Page three, oh okay, the website is secure?
Page four, solutions.
The tester recommended that the WordPress version be updated. However, it was at the most recent version, thus is a null finding. The tester also details an issue in the URL shown above, but provides no background. What they do give, is replacement code for the file:
This is very odd and confusing. But you know what else is confusing, a recommendations page after a solutions page which shows mitigations for totally different findings:
Both points one and two are already satisfied, however point three is a good recommendation.
Overall, this is a bad report. It’s confusing, it’s misleading and the found issues aren’t true. It really does seem like this is all just unverified output from a scanner. Personally, I’d rather go with the $5 tester over this $15 tester, which is an interesting discovery. However, they both have missed essentially all issues and therefore I’d go with neither.
$15 Tester 2
I was genuinely excited to receive this report, I was expecting good things from a tester who quickly found the MySQL issue. However, I received a txt file:
Firstly, this tester detailed that DDOS protections were in place and the website had a lack of HTTPS.
This is a good start as I am using Cloudflare and HTTPS was a finding we wanted. However, a very interesting next line “Not vulnerable to XSS”. I can’t understand why the tester has specifically detailed this and not equally stated ‘Not vulnerable to SQLi, XXE, etc’).
Next up we have the Weak MySQL finding. The tester details the weak password and provides the fix. Disappointingly, they have not recommended we restrict root login on the external service.
Finally, a bit of a random recommendation which is to avoid over usage of Woocommerce. Plenty of WordPress web applications use this for their cart system. It does have old vulnerabilities, however, if you remain up to date, you will be fine.
Overall, I believe this tester was the best of the bunch, but let’s face it, the bar was not high. This tester, as with the other testers, did not find almost all the potential issues and thus a false sense of security is being given to the end user.
Final Thoughts
Quite clearly, low cost security assessments are not the way forward. Obviously, these assessments were the cheapest possible option, but they do mirror the type of assessment you would be looking at should you go with a cheap vendor.
If these freelance testers raised their price in line with the industry and produced a nicer looking report, you would likely think you are getting good testing value, you might be happy as the report looks official. Put two and two together and you might find that you have been in this situation before without realising.
What was the common theme throughout this experiment? Well, every tester relied on the usage of scanners, rather than actual knowledge and manual testing. This led to almost none of the issues being identified other than the obvious lack of HTTPS. A good penetration test involves several methods of testing and what these testers have produced is a vulnerability assessment. If you are looking to assure the security of your platform, this is not the way forward.
To reiterate, “You buy cheap, you buy twice”. It rings true in any industry, product, service and pretty much anything. What helps you find a good penetration tester to prevent this?
Avoid simply going for the low-cost option
I’m not suggesting every vendor on the lower end of the money scale is awful. Nor am I saying high price is the best quality. But you should be questioning their ability and know what to look for in your testing partner. For example:
Accreditations
Accreditations are very important in this industry. They ensure a company is up to the required standard to conduct certain jobs. In the UK we have CREST, CBEST, STAR and many more whom certify companies and make certain their testers are capable of a job. You should be asking what certifications a company and their testers hold and compare this along with the financials.
Reputation & Experience
Another few questions to ask. How long has the company been around? How many employees do they have and what are their backgrounds? What type of clients do they typically assess? Questions like these will give you a good idea of what they have done previously. Another good question to ask is, can I see a sample report? That way you have an idea of how well they present issues.
Documents and Communication
Finally, formalities. Does the company require you to complete scoping forms to assure testing of the correct entity? Do they require an authorisation signature? Do they require a purchase order and how will they invoice you? Will they conduct a pre-engagement call and keep you up to date on high risk findings? These are common business practices in the industry and are important to follow.
By knowing these types of questions above will give a good indication as to who the real deal testers are and who are the cowboy testers.
Community Response
You may think “Why should I listen to this guy” and it’s a fair point. But you don’t just have to take my word on this. I initially conducted this experiment to make a video for YouTube:
Once uploaded, I posted this link onto the /r/security sub-Reddit which caught a large amount of attention with 264 upvotes and 73 comments:
https://www.reddit.com/r/security/comments/favjc3/i_built_a_vulnerable_website_and_hired_three/
This post managed to stay relevant at the top of the page for 2 days. At the time of writing this blog, the video has 4887 views, 133 likes and 25 comments and the response from the Infosec community has been great over several mediums:
To conclude, always remember when you are looking to buy something, that there is usually a reason why it is cheap!
Last updated
