Catzie.net Blog

Category: Coding

  • PHP: Parse Dates in CSV with inconsistent date formats

    A friend was having trouble with a CSV file that she needs to process via PHP before saving to database. The CSV files uploaded to the server have inconsistent date formats such as:

    So I prepared a PHP script that lists possible date formats in an array $possible_date_formats and then iterates through the dates read from the CSV file, matching each against the date formats in the array.

    Assuming you have a .csv file with tab-delimited data such as this:

    My PHP file will output this when it’s run:

    And below is the PHP file I’m talking about.

  • Android app dev: Add ActionBar to Preference Activity

    When I converted an activity into PreferenceActivity, I noticed that its ActionBar disappeared. I’ll show you how I brought the ActionBar back to my PreferenceActivity.

    In this guide I will assume that the name of the file that contains your is SettingsActivity.java, and that the view assigned to your SettingsActivity.java is named .

    First, create settings_toolbar.xml under /app/src/main/res/layout/ and place the following code into it:

    Then add the following overridden method to SettingsActivity.java:

    The Java method programmatically creates a Toolbar object from the XML toolbar at settings_toolbar.xml.

    Reference(s): Creating a Preference Screen with support (v21) Toolbar

  • Install & config Apache, MySQL and PHP on your Mac (MAMP)

    Install & config Apache, MySQL and PHP on your Mac (MAMP)


    There’s a Mac computer that I want to use as local development server for web apps. I don’t want to use bundled installers like XAMPP or MAMP because I feel that I will learn more about Apache if I install each component of the LAMP stack separately. By the way, one of my goals is to be an expert in the LAMP stack. And prior to writing this blog entry, I had only been concerned about the actual coding of web apps, and have never had the chance or need to play with the installation and configuration of Linux, Apache, MySQL and PHP as individual components since there’s always been XAMPP and MAMP for easy setup in my local environment. And at work, someone else is always in charge of setting up our Linux server and the rest of the things we need for our production environment.

    I also plan to setup a LAMP environment on my Mac by using VirtualBox to create a virtual machine with Linux, but I’m saving that for another post if it happens. 😆

    Here in this blog entry, I shall take note of the steps that I take as I setup Apache, MySQL and PHP on Mac.

    My environment

    Computer: MacBook Pro
    Operating System: OS X El Capitan version 10.11.5
    I’ll be using the Terminal and Vim editor

    Apache, MySQL, and PHP versions that I will setup:

    Apache: version 2.4.18 (bundled with El Capitan, default)
    MySQL: version 5.7.15 (installed via Homebrew)
    PHP: version 5.5.34 (bundled with El Capitan, default)

    I have no MAMP, XAMPP or any bundled *AMP stack installed on my Mac, so I will have fresh install of MySQL.

    Start Apache on your Mac

    Apache is already in OS X El Capitan so I just need to set it up. If you want to check your Apache version, run the command httpd -v in Terminal. Mine is Apache version 2.4.18.

    Type this command in Terminal to start Apache on your Mac:

    Now that we started Apache, you should now be able to access the URL http://localhost/.

    Start PHP on your Mac

    Maybe you expected this section to be for MySQL setup, but I’m going to work with PHP first since it should be already in OS X El Capitan, like Apache.

    Also, as of writing, the latest PHP version is 7 but I want to take one little step at a time as I write this so I’ll stick with the PHP 5 bundled with my Mac. I will try to post another blog entry on how to upgrade from PHP 5 to PHP 7. 🙂

    We are going to edit the Apache configuration file to enable PHP on Mac, but for now let’s make a back up of the file. Let’s do that by running these commands:

    Notice the 20160910 in the command sudo cp httpd.conf http.conf.20160910.bak. It’s the date when I made a backup. You may change the date or even delete it.

    Open Vim to edit httpd.conf by running this command:

    Find the following line in httpd.conf and uncomment it:

    Remove the hash symbol (#) at the beginning of that line to uncomment it.

    Vim editor tip: when you open the httpd.conf file, search for the line containing “php5_module” by pressing the esc key to make sure you’re not in INSERT mode, then type /php5_module and then hit the return/enter key. Vim should point you to the line that you should uncomment. Press the “i” key to enter INSERT mode in Vim, and uncomment that line. To save the file, press esc key again and type :wq!

    Now, restart Apache to reflect the change:

    Check if PHP works

    Run these commands:

    Press the “i” key to enter INSERT mode again, then place this PHP code into temp_phpinfo.php:

    Then using your web browser, go to http://localhost/temp_phpinfo.php

    If the page shows a table of details about PHP, then PHP now works on your Mac. Good job! 🙂

    I recommend that you delete this temp_phpinfo.php file though, and avoid using phpinfo() especially in production servers because it can lead to security issues. It will be easier for attackers to hack into your system when they see all the details that phpinfo() exposes.

    Install MySQL

    Install Homebrew if you haven’t yet. I choose to install MySQL with Homebrew because I find the process simple.

    Run this command to install MySQL via Homebrew:

    The installation process on my Mac looked like this:

    By default, Homebrew installs MySQL without root password, and with this setup, you can now log into MySQL by typing:

    But wait! Securing MySQL with a password is highly recommended, and if you agree, run this command:

    With a password for root set, you should access MySQL with this command instead, adding a -p to the command for a password prompt:

    MySQL command line on a Mac Mini with MAMP bundle installed

    The Mac Mini I’m using at work already has MAMP installed, but I followed the steps for setting up Apache, MySQL and PHP in this page just to test. Since MAMP is already installed, I already have MySQL.

    So, before using MySQL via command line, simply start your MySQL server from MAMP and then use command line MySQL by logging in like this:

    Create sample database table for testing with PHP later

    Run the following commands:

    This is what my MySQL command line interface looked like:

    Make sure PHP can communicate with MySQL

    The following command creates a symbolic link to help point to the right socket for MySQL to use on Mac.

    Reason: “You may have a socket (appearing as a zero length file) as /tmp/mysql.sock or /var/mysql/mysql.sock but 1 or more apps is looking in the other location for it.” – source

    Another option: A fellow developer told me that instead of creating a symbolic link to the socket, he connects directly to localhost:3306, connecting through networking sockets instead of Unix sockets.

    Enable mod_rewrite

    My guide suggests we enable this now to avoid forgetting later that it’s not yet enabled. This will prevent hiccups when we’re already busy with coding web apps.

    Edit the config file with Vim.

    Uncomment the following lines:

    Restart Apache:

    Sample PHP-MySQL code to test your “MAMP” stack

    Remember the table I created earlier? This is when we’re going to use that.

    Run this command, we’re going to paste code into it:

    Paste the following PHP code which SELECTs data from a sample MySQL table. Remember to edit the variable values to match your own settings, especially the password.

    Visit this link on your browser: http://localhost/testsql.php

    It should display this:

    1 catzie
    2 petar
    done

    Move Document Root folder

    Right now, the document root is at "/Library/WebServer/Documents", and you may have noticed that each time we edit a file to create a new file within this folder, we need to sudo. That is inefficient, so let’s move our document somewhere that we won’t have to sudo for each change that we want to make.

    We will move document root from "/Library/WebServer/Documents" to /webcontent.

    Create “webcontent” folder under the root directory /. You can choose any other name you like.

    Let’s create a sample page inside the newly created folder.

    And place anything there such as “Hello!”

    Now, let;s edit the Apache configuration file to point to the /webcontent folder:

    Change the line DocumentRoot "/Library/WebServer/Documents" into /webcontent

    Change the line <Directory "/Library/WebServer/Documents"> into <Directory "/webcontent">

    Restart Apache:

    Visit http://localhost again, and you should see whatever text you placed inside /webcontent/index.html

    Just in case you still need to ‘sudo’ to access /webcontent folder

    Try to copy the /webcontent/index.html file to check if you no longer need sudo in the /webcontent folder:

    If it gives you the error cp: /webcontent/index2.html: Permission denied then here are your options:

    • Option #1: Change ownership of /webcontent folder

      In the Mac Mini I use, the directory permissions for /webcontent is drwxr-xr-x which means only the owner (root) has permissions write to it, unless I sudo as another user, which is not convenient if I need to edit many files.

      What we can do is change ownership of /webcontent from “root” user to our own user, like so:

      Now try to create or edit files under the /webcontent folder. Your system should no longer deny you permission, since it is owned by you. So far I have not encountered a problem yet with this setup.

    • Option #2: Move document root under your user folder

      You can try to move your document root under your user folder, i.e. ~/ or /Users/<yourUsername>, and see if you can edit files there without the need for sudo. Your user is the owner of this folder so you shouldn’t need additional permissions to edit files under it.

    On my MacBook Pro, I no longer had to sudo when I edit files in new document root /webcontent, but on another Mac computer that I use (Mac Mini), I still needed sudo to edit files in document root. I will edit this blog entry when I find a way to avoid this.

    Tested on:

    Computer: Mac Mini
    Operating System: OS X El Capitan version 10.11
    Apache: 2.4.16 (bundled with El Capitan)
    PHP: 5.5.27 (bundled with El Capitan)
    MySQL: 5.5.34 (bundled with installed MAMP)

    Extras…

    How to stop Apache on your Mac

    Type this command in Terminal to stop Apache on your Mac:

    or

    Based on my observation, when I start and stop Apache multiple times, the stop command with -k only works on the first time. Succeeding commands for stopping Apache becomes just sudo apachectl stop.

    How to check version of Apache, MySQL and PHP

    Apache version:

    or

    MySQL version:

    If using MAMP bundle, MySQL service should be running when you use this command:

    The welcome screen should show MySQL version when you login to MySQL via command line:

    PHP version:

    ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/tmp/mysql.sock’ (2)

    When I got this error regarding a socket (my MySQL is installed via Homebrew), it was fixed when I started MySQL by running the following command:

    I think the same command is used for both starting and restarting of the MySQL server. Note that this is the command I use for my MySQL that’s installed via Homebrew. If you have MAMP GUI and not the MySQL from Homebrew, simply use that interface to start your MySQL service.

    Reference(s):
    Install MySQL on Mac OSX with Homebrew
    Running Simple Select Statements
    symbolic link setup for correct socket file locations in Mac OS X for MySQL
    Version of Apache in Mac
    Main Guide: Installing Apache, PHP, and MySQL on Mac OS X El Capitan

  • Read & write to NTFS from your Mac computer

    Read & write to NTFS from your Mac computer

    My external HDD is in NTFS format and the Mac I’m using at work cannot write files to it. I solved this issue by running NTFS-3G on my Mac OS X version 10.11.


    Follow these simplified instructions to mount your NTFS drive as writeable in your Mac. If you encounter any problem, scroll down below the instructions to see solutions for errors I encountered.

    1. Install Homebrew package manager by running the following command, as instructed in brew.sh:

    2. Install osxfuse if you haven’t yet.
    3. Install NTFS-3G from Homebrew by typing this on your Terminal:

    4. Plug your external drive into your Mac
    5. Unmount the external drive from your Mac if it’s already mounted. (Unmount, not unplug!)
    6. Run the following commands to manually mount the external HDD as NTFS-3G:

    7. Your external drive should now appear as NTFS-3G format drive, and you should be able to write on it now from your Mac.

    Error Fixing

    OsxfuseRequirement unsatisfied

    I tried to install NTFS-3G on my personal MacBook Pro but it failed with this message after installing Homebrew:

    So I run this command to install osxfuse:

    My brew install homebrew/fuse/ntfs-3g command moved past that issue after that.

    brew link errors

    I got these errors after running the command brew install homebrew/fuse/ntfs-3g again:

    So I ran brew doctor and some of the results were these:

    Warning: The following directories are not writable:
    /usr/local/share/man

    This can happen if you “sudo make install” software that isn’t managed
    by Homebrew. If a formula tries to write a file to this directory, the
    install will fail during the link step.

    You should change the ownership and permissions of these directories.
    back to your user account.
    sudo chown -R $(whoami) /usr/local/share/man

    Warning: You have unlinked kegs in your Cellar
    Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
    those kegs to fail to run properly once built. Run brew link on these:
    ntfs-3g
    pkg-config

    First I ran this command for the permission issue: sudo chown -R $(whoami) /usr/local/share/man

    Then ran this command too: brew link ntfs-3g

    After the linking, brew install homebrew/fuse/ntfs-3g finally installed successfully!

    Failed to access volume /dev/disk1s1

    After running the command sudo /usr/local/bin/ntfs-3g /dev/disk1s1 /Volumes/NTFS -olocal -oallow_other, this error was shown:

    ntfs-3g: Failed to access volume ‘/dev/disk1s1’: No such file or directory

    This means disk1s1, as specified in the command, cannot be found. What you can do is find the IDENTIFIER of your external drive from the table that this command returns:

    Reference(s): NTFS-3G on osxfuse

  • Web Security: How SQL Injection is done


    SQL injection is one of the most common website exploits. For us developers to prevent SQL injection attacks, we must first understand how it’s done. Together, let’s review the basics of SQL injection.

    Remember, let’s use our knowledge on good things only, okay?

    Okay meme

    The scope of this post is only a MySQL injection sample from PHP code. I am on a Windows 7 computer, using Git Bash and running SQL commands via command line, and coding my PHP file via VIM editor. My localhost is run by XAMPP.

    I will be using MySQL, not MySQLi or PDO which have prepared statements, because the SQL injection is easier to demonstrate in MySQL.

    Create database

    I’m running these commands via command line.

    What I do is run the mysql command like this: path_to/xampp/mysql/bin/mysql -h localhost -uMyUsername -pMyPassword


    Then run use MyDatabaseName; to select a database.

    This is a very simple database table with only 3 columns: the primary key “id”, “name”, and “secret”. The value of “secret” column, as its name implies, is user’s secret so nobody else should know about this, especially unauthorized users.

    Then insert data using the following commands:

    As you can see, those are dirty little secrets of our users, which in my case, are dogs in my house. :3

    epi-pinky-asia-dogs-chihuahua

    They are cute little chihuahuas who have embarrassing secrets just like humans. But because I’m using their secrets are samples in my database, these secrets won’t be secrets any longer. I also need to not secure my script for demo purpose. Sorry little babies 🙁 The readers don’t know you personally anyway. Hehehe

    show tables; in mysql command line should show that 3 rows have been inserted.

    Unsecured PHP script that runs MySQL query

    Let’s save this as path_to/xampp/htdocs/test_sql_injection.php

    Expectation vs. Reality

    Expectation: The PHP script should be accessible through this URL: http://localhost/test_sql_injection.php since we saved it in the htdocs folder. We will be able to query the ‘users’ table by supplying a GET param called “name” in the URL, like so: http://localhost/test_sql_injection.php?name=epi and the script will show us the id, name, and secret of the matched row.

    It should be safe as long the attacker only searches for a name that he/she already knows, right? Hmm, nope!

    Reality: If the attacker passes malicious string as GET parameter, he/she can view ALL rows in our users table.

    How? I’ll walk you through it.

    Exploiting the vulnerability of PHP script via SQL Injection

    Note: Originally, my PHP script had $_POST data instead of $_GET and I was POSTing the ‘name’ for the query via curl --data "name=A' OR 'x'='x" http://localhost/catzie_injection.php. After changing it into GET, I couldn’t hack the script with the same string value passed to ‘name’. I don’t know why. So to make life easier, let’s just hack it by accessing the URL via web browser.

    Imagine yourself as a curious person with potential in hacking, and pretend you don’t know what we inserted to the database earlier! 😆

    meme challenge accepted

    Access the following URL with your browser:

    It should should give you information about the user named Epi. Here’s what it shows:

    Connected to db
    your input param is epi
    Your MySQL query is: SELECT * FROM users WHERE name = 'epi'
    id: 1, name: Epi, secret: I eat poop. Don't tell mommy.
    That is all!

    yuck meme

    Oops, we saw Epi’s secret! That’s literally a dirty little secret, isn’t it? But that’s ok, because Epi told us we can look up her secret in this database.

    Now, aren’t you curious about the secret of other users in the database? Aren’t you tempted to hack into the database?

    How can we hack this script so that we could view all database rows, and not just Epi’s details?

    Let’s try this URL, with empty query:

    Uhhh, nope. I only got this:

    Connected to db
    missing name param

    meme-face-thinking

    Ooh, I know! Maybe I can manipulate the query string of the PHP string!!!

    Let’s try this now:

    I got this result in my browser:

    Connected to db
    your input param is x' or 'x'='x
    Your MySQL query is: SELECT * FROM users WHERE name = 'x' or 'x'='x'
    id: 1, name: Epi, secret: I eat poop. Don't tell mommy.
    id: 2, name: Pinky, secret: I'm a spoiled brat who's always fighting with other dogs
    id: 3, name: Asia, secret: I eat too much my belly is touching the floor
    That is all!
    success-meme-face

    Now we know the secrets of other users. 😆

    The danger

    You should understand by now that this vulnerability can lead to exposure of even more sensitive information to the attacker, such as passwords, contact numbers, addresses, etc.

    Explanation of how we hacked to see all database rows

    How did that happen? It’s all in the manipulated query!

    The intention of the PHP script is to construct a query like this:

    But instead of passing a real name, we passed this value:

    Notice that we did not add a quote at the beginning and end of the value. It’s because that value is inserted between the single quotes in my PHP script’s query string, so that it produces this query:

    The 'x'='x' in the WHERE clause will always return true, and since “or” only requires one true value, then every row in the database becomes a match for the SELECT query.

    How to prevent SQL injection attacks like this?

    Sorry but I won’t cover tutorials on how to do that in this post.

    meme are you kidding me

    Now, now, don’t get mad! Basically, what you need to do is use “prepared statements” with MySQLi or PDO.

    Please refer to the following articles that will give you precise instructions on how to secure your PHP scripts and web apps so that you could prevent SQL injection attacks.

  • Prevent direct access to directories of subdomains

    From my hosting platform (e.g. cPanel), I created two subdomains for web apps.

    The web apps reside in the following directories:
    /foo/bar/app_one/
    /foo/bar/app_two/

    And the subdomains I created are accessed through these URLs:
    http://app_one.bar.domain.com (pointing to /foo/bar/app_one/ directory)
    http://app_two.bar.domain.com (pointing to /foo/bar/app_two/ directory)

    For security purpose, I don’t want anyone to access the web app directories via these URLs:
    http://domain.com/foo/bar/app_one/
    http://domain.com/foo/bar/app_two/

    So, I updated the .htaccess in my public_html and added this:

    With these lines added, anyone who tries to go to http://domain.com/foo/bar/app_one/ is redirected to http://app_one.bar.domain.com

    Reference/Source: Prevent Access to Subdomain Folder from Main Domain

  • Learning MongoDB on Windows 7 (Shell Edition)

    I’ve worked on a high-volume site that processed thousands of telecom network subscribers everyday. Me and the rest of the team managed to handle the challenges using MySQl and query optimizations, but now, I wonder how easier the tasks would have been if we had used NoSQL / MongoDB back then.

    My environment and methods:

    As I wrote this tutorial, I was using Windows 7. I will be using MongoDB Shell edition on my Git Bash.

    I actually haven’t used MongoDB yet, but I’m trying to learn it now. Let me share with you the steps I’ve been taking.

    MongoDB: Beginner’s guide

    For the MongoDB terms such as document, collection, shards, query routes, All out beginner’s guide to MongoDB from Analytics Vidhya provides good explanation.

    Download for Windows 7

    Download the latest version of signed .msi file from MongoDB Win-32 downloads page. As of writing this was the version I downloaded.

    Getting Started with MongoDB (MongoDB Shell Edition)

    Let’s follow the instructions from the MongoDB website on getting started with MongoDB using Shell.

    You may choose other editions such as Java, Python, Node.JS, and others in this page.

    Common hiccups

    MongoDB error: SyntaxError: missing ; before statement @(shell)

    While running the mongoimport command, if you get this kind of error in your shell for MongoDB:

    E QUERY [thread1] SyntaxError: missing ; before statement @(shell)

    Try to open a new command prompt / Git Bash window (or whatever you’re using), navigate to the directory where your mongo.exe is (e.g. C:/MongoDB/bin/) and run your mongoimport command there.

    MongoDB Improper shutdown

    When I press Ctrl + C it leads to “improper shutdown” error. Refer to this doc to repair / recover from improper shutdown of MongoDB.

  • Laravel 5: create route group that is only available during testing (Middleware)


    Today I needed to protect a group of routes from everything except my testing environment. I need to run PHPUnit tests that depend on certain routes, but I don’t want those routes to be available anywhere else. I’ll share in this blog post what I did to get what I need.

    Assumptions in this tutorial:

    You already have Laravel 5 installed on your computer and you are running the app in “testing” mode (e.g. PHPUnit testing)

    Introduction

    In Laravel, Middleware filters requests that are coming into our application. One of the pre-made Middleware in Laravel 5.2 is Authenticate, which looks like this:

    Inside the handle() method, it checks if the request is using AJAX -or- is asking for JSON, and if either condition returns TRUE, it blocks access and returns ‘Unauthorized’ as response. Otherwise, client is redirected to the ‘login’ route, which should happen when client is requesting via web browser.

    What I did to make my grouped routes available only in testing environment (in my case, when running PHPUnit) was create a new Middlewave. I’ll show you how. 🙂

    Create a new file under Http/Middleware and name it TestingEnvOnly.php

    In the handle method, we check if app environment is set to testing, and if so, we allow the request to access the requested route. Otherwise, the request should fail and repond with ‘Unavailable’.

    Now, let’s register this Middleware to Http/Middleware. Add the following to the $routeMiddleware array:

    It’s time to add a testing-only route:

    If you use PHPUnit like I do, you can test our new Middleware by creating this test method:

    I’m only using var_dump so that we can easily see the printed string response which should say “This should be visible only for testing environment” when you run PHPUnit.

    Now, in your web browser, visit the route your set for /api/v1/testing_env_only. It should say ‘Unauthorized’.

    Here’s how the files I edited in Laravel look like now:

    I’m using Laravel 5.2 and these are codes I uploaded at Gist

  • Laravel 5: activate API token guard and print “Unauthorized” instead of login redirect

    In Laravel 5, I wrap my API routes in a route group like this:

    Notice that I use auth:api and throttle on my middleware array.

    auth:api turns on the auth middleware with API token as guard, while throttle activates rate limiter for APIs.

    To get started with API token authentication, you may read https://gistlog.co/JacobBennett/090369fbab0b31130b51.

    I followed the same tutorial to set mine up, but had this issue: Upon using invalid api_token or no token at all, I am redirected to the /login route, which in my case is a currently nonexistent route.

    As instructed by the guide, I already edited my app/Http/Middleware/Authenticate.php file to add the || $request->wantsJson() condition, but my API routes still redirected to /login route upon detection of invalid api_token.

    The solution

    What I did was add another condition: || $guard == "api"

    And now, my app/Http/Middleware/Authenticate.php file looks like this:

  • Android app dev: ShareActionProvider in Activity & Fragment

    I’ve been taking the Developing Android Apps course on Udacity, and yesterday I encountered a difficulty in solving the Quiz: Share Intent.

    One of the course’s goal is to build the Sunshine App – an Android app with weather forecast.

    In that quiz I had to implement ShareActionProvider on my DetailsFragment so that a share icon becomes available on my app’s ActionBar.

    One of the main difficulties was the error that stated ActionProvider cannot be cast into ShareActionProvider. And at some point there was also an error saying that the methods/classes I’m using require API 14, while my minimum supported API is 9. The Udacity course required that I set it to 9 so I couldn’t change that.

    Now, the sharing in my app is working file, and I will share how I did it via Gist. I actually ended up taking a peek at the code provided by Developing Android Apps course on Udacity because I didn’t want to waste any more time doing trial-and-error (I have another project to work on! lol), but I still had to edit their code for ShareActionProvider here and there to make it bug-free.

    And so, take a look at how my Sunshine app codes ended up: