The client required some fairly involved reporting facilities, here is a sample of some of the requirements from the client:

• Aggregated report, split between two dates of our choosing, with totals split by either month, day or week (we want to choose)
• A list of invoices for a day of our choosing with aggregated totals of for the day selected
• Annual invoicing report – we want to show a summary of the total for the items sold, the total vat charged and summaries for average invoice values
• Invoices totals grouped by customer

There were several more, but they were all similar to the above.

Firstly, I’ll explain the database setup, for the invoicing, of the current system I built. It was basically two tables – one for the invoivce header and one table for the invoices items – nothing un typical there at all.

My first point of call to integrate the reporting was to use what I had already. This proved qwuite complicated as it involved fecthing large sets of data to calculate the totals required by the reports – this was also pretty slow when running the more intensive reports over a longer date range. It would get even more complicated and database intensive when I wanted totals for the other reports.

After some consideration, my solution to this was simple. I decided to add a few new columns to my invoice header table – “invoice_total”, “invoice_sub_total”, “invoice_vat” and “total_invoice_items”. For the reporting side of things, this made the whole process much easier. For example:

To get an aggreate total figure invoiced between two dates, I required a single sql statement, that required no joins, or loops in PHP:

SELECT SUM(`invoice_total`) As total_invoiced FROM `invoice_header` WHERE (`invoice_issue_date` >= :myDate AND `invoice_issue_date`

Equally as simple, to get a monthly total between two dates, grouped by month, the sql statement was just as simple:

 SELECT SUM(`invoice_total`) As total_invoiced FROM `invoice_header` WHERE (`invoice_issue_date` >= :myDate AND `invoice_issue_date`

Getting average invoice sub total between two dates:

 SELECT AVG(`invoice_sub_total`) As average_invoice_subtotal FROM `invoice_header` WHERE (`invoice_issue_date` >= :myDate AND `invoice_issue_date` 

The latter made the actual PHP code required to produce the required reports very straightforward – before I would have required joins to caluclate the totals and multiple sql queries – that could have got quite messy. I also gained a few added benefits of storing the totals within my header table:

• On the print invoice screen, there was need to calculate anything, as I simply read the totals from the datbase
• On the invoice listing screens, I could now display an invoice total in a seperate column, as this was again, simply a database value – before I would have needed to calculate this total via the invoice items table

However, this simplicity in one area, came at a cost … unfortunately. As all report data was now based from totals in the invoice header table, this figure needed to be correct at all times. I needed to amend the add and edit invoice functions. When adding an invoice, I needed another loop to calculate the aggregate totals to insert into the database. When editing the invoice, I needed to recalculate these totals each time the invoice was edited (in case a row was deleted for example). Whilst the latter added a little more work when adding/editing invoices, I feel that this is well worth it due to simplicity when dealing with the reporting side.

However, after an hour or so of emailing the store owner I got a very worried phone call. In short, the bulk price update to had caused all archived order invoices to become incorrect. Why – due to bad database design by the original developer, let me explain.

The web developer had stored the order as you would expect – an order header table, coupled with a order lines table (one record for each order item). All good so far. Where the previous web developer had fallen short is not storing the price at the time of the order, instead simply storing the product id within the order lines table and looking order details via the way of an INNER JOIN – bad move unfortunately. Obviously, in any sort of product price update (as above) all of his historical order information would become null and void. The latter was especially bad in this case, as the store owner pretty much used the list of store orders as base for financial and year end reporting, as the business was pretty small (no Sage then). Explaining the latter to a none technical client is even harder, believe me

To avoid such nightmares never, ever store such information in this manner – always store product details at the time of ordering. Storing products details at this stage will ensure that an order made in 2008 will be correct when viewed even 10 years later when prices will have been updated. So, in the order lines table, as an absolute minimum store the product name, price and any attributes that will affect price – meaning an INNER JOIN with the order header table is not required.

Depending upon your business rules and frequency by which your product information changes, you could also store the product at time of ording, within a product archive table – that stores basic product info. This way, there is the benefit of saving the description the customer saw at time of ordering.

What is descriobed above is basically bog standard when making any ecommerce solution, or even using something bigger like Opencart, but when it is not present, the amount of issues it causes is huge.

Luckily for me (and the store owner) I created a quick backup of the entire store’s database schema and data before making the bulk update, so was able to restore this for the time being. All I have to do now is amened the database design accordingly and alter the way the site stores order information.

So, never ever rely upon relational data, to view and store data that needs to be achived.

Full File Level Caching

Full file caching is the simplist method and essentially takes an image of the page at a particular time. This method is very fast to implement but does have it’s downsides. As you are storing a static file of the page you’ll instantly notice the following issues (I’ve included some typical examples you’ll run into):

• Any elements of your page that require session data will fail to work (you’ve essentially cached the session id too!)
• If the page needs to be served using a particular header you’ll be out of luck, as a statiuc file will always be served using a plain text header (E.g. a cached rss feed won’t validate as the header has changed)
• Clearing the cache becomes very clumsey as the whole cache requires cleaning when even a small change is made (E.g. say a new article is added by the admin)
• User logins that make use of session data
• E-Commerce sites and a shopping basket summary
• Search results pages

With full file caching dynamic content will always be a issue. However, full file caching CAN help high traffic sites, even when a short cache lifetime is set – use with caution!

Block Level or Partial File Caching

Block level caching is likely to be a much longer term caching solution. It is more likely that you’ll have certain sections of a site that do not require an up to date minute. For example, say we have an ecommerce store with dropdown menu that list our full category structure from a database – this is likely to be highly database intensive and may use recursion. This output can be cached to a flat file as follows:

$options = array(
    'cacheDir' => '/cached/',
    'lifeTime' => 1000
);

$cache = new Cache_Lite_Output($options);

if (!$cache->start('left_category_structure')) {
   $store->list_full_category_structure();
   $cache->end();
}

This is much better than full file caching for several reasons:

• The page in question can remain dynamic dynamic elements such as the user’s login status on another part of the page
• If a change is made to the category structure I can delete that particular section of the cache only (the rest of the cache does not require regenerating!)
• The cache lifetime can be set on a per block basis – useful for different types of content
• Even if the category structure doesn’t change often, the block could still be cache forever (set cacheLifeTime to null in the options array) and cleaned when a change is made

Taking Partial Caching Further

PEAR Cache_LIte is also fairly flexible and you can make it fit your needs. For example, say we wanted to cache 10 random articles on a website – which consists of a database query that fetches 10 random articles from a database. The idea here is to still cache the content in a block as above. However, for this our system would create a number of cache blocks and then service those cached blocks to the user. The solution is a simple one, building on the example from above:

$options = array(
    'cacheDir' => '/cached/',
    'lifeTime' => 1000
);

$cache = new Cache_Lite_Output($options);

if (!$cache->start( 'left_category_structure . rand(1,5))) {
   $store->list_full_category_structure();
   $cache->end();
}

When the user requests a page, 5 different cache blocks will be checked – we are essentially creating 5 different versions of the random cached content.

Story such data need to not be overly complicated. The following, simple, table structure is required (in a real ecommerce system you would definately have additional fields – these have been omitted here for thae sake of the example):

Product
product_id (PK)
name

Category
category_id (PK)
name

Products_Category
product_id (PK)
category_id (PK)

The products_category table is a simple linking table that allows a many-to-many relationship between the product and category table. It contains to two primary keys to ensure every combination of product and category is unique. for example, this table will contain many unique number pairs and a row may be 1,4 or product_id 1 and category_id 4. The files to create and populate this table structure with sample data can be found here.

Now it is simplay a case of running a series of MySQL statements (I’d advise converting them to stored procedures for more security and better application seperation) to retreieve the appropriate data. For example:

Products Within a Certain Category (E.g. category_id 1):

SELECT p.product_id, p.name FROM product p
INNER JOIN product_category pc
ON p.product_id = pc.product_id
WHERE pc.category_id = '1';

Count Products Within a Certain Category (E.g. category_id 1):

SELECT COUNT(p.product_id) As myCount FROM product p
INNER JOIN product_category pc
ON p.product_id = pc.product_id
WHERE pc.category_id = '1';

…and that’s it. Extremely simple, can be expanded to any eCommerce system and not convoluted at all

For the breadcrumb naviagtion I needed a function to displaying the path to a given category ID, sometimes refered to as a single branch or node. I was using a simple parent child database table structure:

While the latter may seem fairly trival I really couldn’t get my head around the problem. After a bit of Googling, I found a function that was a good starting point so adapted it to fit my problem (the function is part of a categoru class):

function getCategoryTreeIDs($catID) {
		$row = DB::getInstance()->query("SELECT parent FROM categories WHERE ID = '$catID'")->fetch();
		$path = array();
		if (!$row['parent'] == '') {
			$path[] = $row['parent'];
			$path = array_merge($this->getCategoryTreeIDs($row['parent']), $path);
		}
		return $path;
	}

The function simply returns an array of category IDs. E.g 20, 28. So from the array I’d know that the tree would go as Home > Cat ID 20 > Cat ID 28.

Displaying the Breadcrumb Navigation

To display the actual breadcrumb I simply added the following method, that loops through the array of ID we just generated. The getNameLink method simply generates an SEO feindly website URL for the category, inside the <a> tag.

function showCatBreadCrumb($catID) {

		$array = $this->getCategoryTreeIDs($catID);

		$numItems = count($array);
		for ($i = 0; $i<=$numItems-1; $i++) {
			echo $this->getNameLink($array[$i]) . ' &raquo; ';
		}
	}

The result is a nicely formatted breadcrumb (to use our tshiorts example again):

Home &rquao; Clothes &rquao; tshirts &rquao; Mens &rquao; Red tshirts &rquao; Offensive &rquao;

A recursive function inside a loop, are you insane?

Some of you may have noticed that the function used to generate the category IDs is called recursively. This generally considered bad practice, for large data sets due to performance issues. However, for the current use this isn’t an issue. I know for a fact that client won’t be adding categories more than several levels deep, so performance really isn’t an issue in my eyes here. Maybe if we had hundreds of categories, but for several it’s really a non issue in my opinion.

getCategoryTreeIDs

Page rank	5

Page rank	5

Page rank	5

Page rank	5

What follows is a list of some very useful MySQL queries for use in projects, enjoy. I won’t go into detail how and why they work, I’ll leave that up to you

/*** select records from the previous day ***/
SELECT * FROM users WHERE TO_DAYS(last_login) = ( TO_DAYS(NOW()) - 1 )

/*** select records from last 90 minutes ***/
SELECT* FROM users DATE_SUB(NOW(),INTERVAL 90 MINUTE);

/*** select records from last 1hr 5 mins ***/
SELECT DATE_ADD('1970-01-01 12:00:00', INTERVAL CAST(6/4 AS DECIMAL(3,1)) HOUR_MINUTE);

/*** select records from last hour ***/
select DATE_SUB(NOW(), INTERVAL 1 HOUR);

/*** using the SIGN function to mark a number as positive, negative or null ***/
SELECT backlist, SIGN(backlist) AS user_to_backlist
FROM users
WHERE user_banned IS NOT NULL;

/*** select records from last week ***/
select DATE_SUB(NOW(), INTERVAL 1 WEEK);

/*** get the last day of next month ***/
SELECT LAST_DAY('2006-03-06' + INTERVAL 1 MONTH) AS last_day;

/*** select unique records only ***/
SELECT user_name FROM users GROUP BY users HAVING ( COUNT(user_name) = 1 );

/*** select records from one table that are in another table i.e. all the customers that have placed an order ***/
SELECT DISTINCT cust.customer_id, cust.customer_name
FROM cust INNER JOIN orders ON cust.customer_id = orders.customer_id;

/*** insert data from one table into another ***/
INSERT INTO customers(customer_id, customer_name)
SELECT cus_key, cus_name
FROM customers_2 WHERE customer_name LIKE 'W%';

/*** update information based upon a seperate table ***/
UPDATE cust SET status = '1'
FROM orders WHERE orderdate > '2009-01-01' and orders.customer_id = cus.customer_id;

/*** classic self join example - who is an emoployees manager ***/
SELECT emp.empID, emp.Name, emp.Salary, managers.Name AS manager_name
FROM emp
LEFT JOIN emp AS manager_name
ON emp.ManagerID = Manager.EmployeeID
WHERE (emp.empID= '123456');

/*** using UNION to combine results from multiple queries into a single table ***/
SELECT users.name
FROM users WHERE (users.name BETWEEN 'A%' AND 'M%')
UNION
SELECT banned_users.name FROM banned_users
WHERE (banned_users.name BETWEEN 'A%' AND 'M%');

/*** concatenate column data into a single column ***/
SELECT CONCAT(emp.firstname, '-', emp.lastname) AS emp_full_name FROM emp;

/*** select count of records for each hour ***/
SELECT HOUR(last_login) AS last_login_hour, COUNT(*) AS the_count FROM users GROUP BY HOUR(last_login);

/*** import a csv file ***/
LOAD DATA INFILE '/path/xxx.csv' INTO users_table csv_test_table FIELDS TERMINATED BY ',' LINES TERMINATED BY "\n" (user_name, access_level , user_email);

/*** display current mysql user ***/
SELECT USER();

<script src="js/jquery.js" type="text/javascript"></script>
<h1>New User Signup</h1>
<form action="index.php" method="post">

  <label for="name">Username:</label>
  <input id="name" name="name" type="text" />
  <label for="name">Password:</label>
  <input id="password" name="password" type="text" />

   <div id="container">
      <a href="#"><span>» Add your favourite links.....</span></a>
   </div>

   <input id="go" class="btn" name="btnSubmit" type="submit" value="Signup" />
</form>

The only part that isn’t standard is highlighted above. This is simply the link users click to add additional form fields on the fly. To make that happen we’ll need some JQuery:

var count = 0;
$(function(){
	$('p#add_field').click(function(){
		count += 1;
		$('#container').append('<strong>Link #' + count + '</strong>'+ '<input id="field_' + count + '" name="fields[]' + '" type="text" />' );
	});
});

Jquery makes it very easy to add for elements using the append method. As we need to assign a unique name to each additional input (as we are allowing the user to add an indefinite amount of inputs) I’ll use a variable to keep track of the number of inputs added. Whenever an input is added, the counter variable increments by one. Another thing to notice is how the fields are named using ‘txt[]‘. This creates an array of field names. So now when we click the ‘add_field’ link, a new input field is appended to the contianer div – view example.

The only other order of business is to save this information to a database. This would be easy if we had static field names for the whole form, but we don’t. We have the following:

<input id="name" name="name" type="text" />
<input id="password" name="password" type="text" />
<input id="field_1" name="fields[]" type="text" />
<input id="field_2" name="fields[]" type="text" />
<input id="field_3" name="fields[]" type="text" />
.
.
.
etc.

To capture this dynamic fields we’ll need to use a bit of PHP to loop through our posted array and saving the data using a many-many relationships using the following database structure – allowing a user to be associated with an unlimited number of websites:

Many-Many Data Structure For Users And Sites

The following PHP code will grab our static and dynamic form values and insert into the above data structure. We’ll be using a class for the database to minimise the amount of code written. Firstly we’ll check if the forma has actually been submitted, grab the static values (username and password) and insert these into our users table:

//If form was submitted
if (isset($_POST['btnSubmit'])) {

	//create instance of database class
	$db = new mysqldb();
	$db->select_db();

	//Insert static values into users table
	$sql_user = sprintf("INSERT INTO users (Username, Password) VALUES ('%s','%s')",
						mysql_real_escape_string($_POST['name']),
						mysql_real_escape_string($_POST['password']) );
	$result_user = $db->query($sql_user);

PHP’s sprintf function was used to format the sql query. I also escape input to prevent nasty things from happening. Now we need to check if the user has actually added any favourite websites in order to prevent a php error if we try to find non-existant values, if they have added a field we’ll need the user id number previously inserted into the users table, as we’ll need to insert this elsewhere later on:

//Check if user has actually added additional fields to prevent a php error
	if ($_POST['fields']) {

		//get last inserted userid
		$inserted_user_id = $db->last_insert_id();

Next we loop through the added form field values. This is where naming the field as in our append function as ‘txt[]‘ comes in, as all fields we added are submiited as an array called txt. Using a for each loop we can assign values to all dynamically added fields and save them to database. The $key is simply the fields name (field_1) and the $key is our actual value entered for that field. Now we know this, we can easily generate the sql statements within the loop. Firstly we add the website into the websites table, get the id of the inserted website and insert this into the the link table, using the userid from before. To finish, we display a confirmation message and disconnect from the database:

//Loop through added fields
		foreach ( $_POST['fields'] as $key=>$value ) {

			//Insert into websites table
			$sql_website = sprintf("INSERT INTO websites (Website_URL) VALUES ('%s')",
						    	   mysql_real_escape_string($value) );
			$result_website = $db->query($sql_website);
			$inserted_website_id = $db->last_insert_id();

			//Insert into users_websites_link table
			$sql_users_website = sprintf("INSERT INTO users_websites_link (UserID, WebsiteID) VALUES ('%s','%s')",
						    	   mysql_real_escape_string($inserted_user_id),
								   mysql_real_escape_string($inserted_website_id) );
			$result_users_website = $db->query($sql_users_website);

		}

	} else {

		//No additional fields added by user

	}
	echo "
<h1>User Added, <strong>" . count($_POST['fields']) . "</strong> website(s) added for this user!</h1>
";

	//disconnect mysql connection
	$db-&gt;kill();

As a result, our database should be populated as follows (these are screenshots from phpmyadmin):

All our data tables are populated and the correct website and user id are inserted into the link table

You can view the source of php file here or download the original source files (you’ll need to open classes/db.class.php and populate the sql variables with your own database details). The MySQL source to create the tables is also included in the source files.

UK Business Directory