Backup and Migrate - Restore 1 Table

Well I never thought about trying this before, and there was no reason why it wouldn't work, but restoring a single table into a Drupal website using the Backup and Migrate module worked like a charm!

Scenario

I'm using the IP Ranges module on several websites across several servers. Most of the time the websites have content that is specific to certain geographic regions or countries, so getting hit a ton by China, Russia, or Brazil isn't desirable.

Also, adding the IP Ranges entries to each website (or even a new one) is tedious.

Enter Backup and Migrate

On one site, I do all of my manual entries using the IP Ranges interface. Then I can do a backup of the IP Ranges table from this site. The backup will contain info for just IP_RANGES and nothing else. I do not want to break my other sites. I have a Settings Profile for IP Ranges and then I do a manual backup.

On another site, do a restore using that backup file. But wait.....read below.

One Gotcha!

The backup contains a "DROP TABLE IF EXISTS" statement for ip_ranges. Now the script should be fairly quick, but dropping the table while the module is enabled can cause issues for the site if someone visits.

What I like to do is replace the statement in the script with "TRUNCATE" giving: TRUNCATE `ip_ranges`;

double check the rest of the script to make sure you haven't accidentally included other stuff.

Then run the script through the rest of the sites!

I'd rather use Firewall rules, but I need to do more research on that. I've broken access to sites before because I've entered the wrong Firewall rule. :P

Blocking the Spammers

Stop the spammers and bad referrers.

This is a compilation of info I found on various sites.

Add this to the end of your .htaccess file:

SetEnvIfNoCase Via evil-spam-proxy spammer=yes
SetEnvIfNoCase Referer evil-spam-domain.com spammer=yes
SetEnvIfNoCase Referer evil-spam-keyword spammer=yes
SetEnvIfNoCase Via pinappleproxy spammer=yes
SetEnvIfNoCase Referer semalt.com spammer=yes
SetEnvIfNoCase Referer semalt.semalt.com spammer=yes
SetEnvIfNoCase Referer poker spammer=yes

Order allow,deny
Allow from all
Deny from env=spammer

Drupal Modules I use:

ip_ranges - https://www.drupal.org/project/ip_ranges - Block entire range of IP's.

Additional Sites:

http://myip.ms/info/whois/  OR any other popular Whois database. I use this to see the range of IP's I want to exclude. 

This process is a little long for me. Going through the logs and seeing what IP's are hitting bad URL's or protected URL's. Then find the IP range and enter into IP Ranges.  I'd like to see a nice report that uses a Whois database to show where the requests are coming from.

I'm still looking for a better overall solution. For example, I've got to be able to put the rules somewhere in my Apache config to cover all the sites on my server, rather than configuring each one individually, I just haven't taken the time to do it yet. We'll see. :)

Module - Front Page

Do you want different user roles to have different Front pages when they login or go to the Home page?

The Front Page module (https://drupal.org/project/front) works very well for that.  18,109 reported installs at the time of this writing.

If you want token support, 7.x-2.4 and lower will need a patch: https://drupal.org/node/1786128

Front Page handles roles in the reverse order of what you have defined in the Role arrangement and creation page (admin/people/permissions/roles), so Admin role, then Authenticated, and then finally Anonymous role will be processed when implementing a custom front page.

Here's a screenshot of 2 roles, Authenticated and Anonymous.

As an example, for an e-commerce site and with token support, I can redirect a "customer" role to their Orders page.

More advanced users and cases will probably want to use Rules.

Handle potential plural items

Let's say you have a count of items ("widgets") that you want to display.

When you count them, the result could be 0, 1, or more than 1 item.  SO you need to display one of the following:

• 0 widgets
• 1 widget
• 2 widgets

In PHP, you can handle this with the following code:

<?php print $view->total_rows;
 if ($view->total_rows == 1) {
   print " widget"; 
} else {
   print " widgets";
}
 ?> 


Or use Drupals built in function "format_plural" to help out:
<?php print format_plural(max(0, $view->total_rows), '1 widget', '@count widgets'); ?>

JSON for Drupal select list and location

Background

I'm using PhoneGap and DrupalGap to create my first Android app called "2 For Deals" which helps you track and locate deals in stores that are "2 For"  some dollar value.  Location of the item, as well as expiration of the deal are tracked.  Have a gander (with screenshots): https://play.google.com/store/apps/details?id=com.twofordeals

Tech Talk

DrupalGap provides a JavaScript library for your PhoneGap app, as well as a Drupal module which provides services for accessing data through the Drupal site/data.  Great stuff if you want to combine the two.

JSON

Here is the JSON I used to create a new node which includes the following fields:

• field_cost: A Decimal field type using the text field widget
• field_select:  A "List (text)" field type using the "Select list" widget.  Notice the difference between this and some of the other fields.  No index is used.
• A simple Location determined by users' GPS.  This is the location for the node, where 1 and only 1 is allowed.  Not sure if this is correct, but I had to provide at a minimum the locations and location fields.
• Body:  standard body field used in Drupal

{
"node": {
"type": "my-content-type",
"title": "This is my new title ",
"language": "und",
"field_cost": {
"und": {
"0": {
"value": "5"
}
}
},
"field_select": {
"und": {
"value": "key-from-drupal for the select"
}
},
"locations": [{
"locpick": {
"user_latitude": "41.98464",
"user_longitude": "-91.673859"
}
}],
"location": {
"locpick": {
"user_latitude": "41.98464",
"user_longitude": "-91.673859"
}
},
"body": {
"und": {
"0": {
"value": "This is the body of my node"
}
}
}
}
}

For blogging purposes, some of the fields are made more generic. The location is one that exists, but points to the middle of a river.  You won't find me there.  :)

I'm posting this because different pieces caused me different headaches at times.  For example on the mobile app, I wanted to collect as little data as possible about the location, so I needed to find the sweet spot of how to create the location on the server side with as few pieces of data as possible.

TODO:  I will eventually be changing the Location to a node reference.  Which means I will need to find the location(s) closest to the users location, display a list, then submit using that reference.  Fun!!!

I hope this helps someone.  Enjoy!

New Server setup with CentOS 6, Drush and Drupal libs

My compilation of docs and tutorials to help me get a server setup with CentOS 6, Drush and various libs needed for Drupal 7.  Once that is complete and working, a short write-up on getting a Drupal 7 site up and running in less than 2 minutes.

1. Webmin install and setup: http://jarodms-drupal.blogspot.com/2012/07/webmin-on-ec2.html
2. Config Firewall for Webmin:   http://www.webmin.com/firewall.html
3. LAMP Setup: http://library.linode.com/lamp-guides/centos-6
4. Drush in less than a minute: http://danreb.com/content/how-install-drush-centos-linux-drupal-development
5. Additional libraries for Drupal:
1. yum install php-gd
2. yum install php-dom
3. yum install php-mbstring

Additional Steps:

1. Optimize php.ini
2. Depending on server, may need to open firewall for port 80

Install Drupal 7 instance in less than 2 minutes:

1. Create MySQL Database
2. Create MySQL User/pass
3. Grant MySQL privileges for user
4. Config virtual host for new Drupal site and restart Apache

<VirtualHost *:80>

     DocumentRoot "/var/www/html/drupal7"
     ServerName fill-in
     ServerAlias fill-in
     ErrorLog /var/www/logs/drupal7-error.log
     CustomLog /var/www/logs/drupal7-access.log combined
     <Directory "/var/www/html/drupal7">

RewriteEngine on

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/favicon.ico
RewriteRule ^(.*)$ index.php?q=$1 [L,QSA]

</Directory>

</VirtualHost>

Use Drush to download Drupal and install site:

1. drush dl 
2. drush site-install standard --account-name=admin --account-pass=admin --db-url=mysql://SiteMySQLUser:SiteMySQLUserPassword@localhost/SiteMySQLDatabase

My Dell Inspiron 1764 - CentOS 6.3 and Broadcom Wireless BCM43224 driver

There is no default driver for Broadcom wireless cards on CentOS 6.3.  Broadcom is apparently getting better at Linux support, so they provide the source you can compile to get the driver.  Here's the awesome and simple writeup that I found.  It really did take just 5-10 minutes of copy/paste/execute.

I won't even try to recreate all the steps in this blog.  Just go to http://elrepo.org/tiki/wl-kmod and follow the instructions.  Worked like a charm!

My setup:
Dell Inspiron 1764 dual-boot into Windows 7 and CentOS 6.3
PCI Device (lspci at terminal) :
Network controller: Broadcom Corporation BCM43224 802.11a/b/g/n (rev 01)

Drupal User Admin module stack

Here's a stack of modules that I like which help a non-administrator manage their own users.

Administer Users By Role:  http://drupal.org/project/administerusersbyrole
Role Delegation:  http://drupal.org/project/role_delegation
Login One Time:  http://drupal.org/project/login_one_time
No Current Password:  http://drupal.org/project/nocurrent_pass

Once installed and enabled, I create a role ("site-owner" for example) and assign it the various permissions.
A few more little tricks and now the Site-Owner can create or cancel users, assign various roles, and send them a One Time Login link if needed.

One caveat I need to investigate...the Administrators (e.g., User 1) are still shown on the list of users (admin/people) and certain roles can be added or removed, but the Administrator role can't be removed and the Admin account can't be edited or canceled.  Really the Site Owner can't do anything harmful to the Admin, but I'd prefer that account wasn't shown at all in the list.  I need a module that will only show the users whose role is less than or equal to the current role.

Drupal 7, Services 3.x , Request and Response

Here is something that I just recently learned regarding request/response from a REST server.
In this case, I am using Drupal 7 with Services 3.x.

It's just one of those things I guess you don't lean until you need to, and I never had yet.  It was difficult to find the info though.

Here is the scenario:

1. You are calling a web service using JSON as the Content Type.
2. You are getting back an XML response.
3. You want to get back a JSON response.

Here's what you need to do.

Goto your services

 

Select "Edit Server"

Make sure the response formatter "json" is selected.  It may already be.

Now in your JSON REQUEST, add the following header:

Accept = "application/json"

For some reason, Drupal was returning the response in XML format by default, I don't know why, but adding this header to the request immediately gave me back a JSON response.

Webmin on EC2

Installing the RPM package of Webmin on EC2 was a 5 step process for me:

*** Verify the latest RPM version at http://www.webmin.com/download.html

1. wget http://prdownloads.sourceforge.net/webadmin/webmin-1.610-1.noarch.rpm
2. sudo rpm -U webmin-1.610-1.noarch.rpm
3. sudo /usr/libexec/webmin/changepass.pl /etc/webmin root foo
1. Change the webmin root password to foo. 
4. sudo /etc/init.d/webmin restart
5. Configure IPTables to allow port 10000: http://www.webmin.com/firewall.html

Webmin will be accessible at http://youserverip:10000

* Works fine on CentOS in Linode and Rackspace.

* Would be nice to install cert for this though.

What's New in Drupal 7

What's New in Drupal 7 | drupal.org

A good page to read...just in case someone asks you.  ;)

'via Blog this'

Getting Drupal updates into TEST

.

1. I create a branch from TEST and call it something like updates. (git checkout -b updates)
1. Now I'm working in the branch updates
2. Create a backup of the database - important and discussed later.
3. Pull the necessary modules or Drupal updates and test it out.
4. Everything look ok?  Then merge the branch to TEST.  
1. git checkout test
2. git merge updates
3. handle any conflicts

Why would you do this rather than pull your Drupal updates right into TEST? Murphy's Law.  If you pull the Drupal updates into TEST first, then somebody is bound to have an issue or item that needs worked on that needs to go into TEST before you had a chance to completely test and verify your Drupal upgrade.  

The approach above helps you get around this.  You're in the middle of testing Drupal updates and someone approaches you with a bug or change that needs to go into TEST first. How do you roll back to the TEST branch? 

1. checkout TEST (git checkout test)
2. Restore the database using the backup you created earlier.
3. You are now at the TEST version you need for the new change, without those Drupal updates that you haven't completely tested.
4. Create your new bug/change branch and continue on the issue that was asked of you.

Drupal - Bootstrap for old PHP script

In a hurry and I have an old PHP application that pulls data from proprietary tables.  I have connected some of the data to a node, but now I need to display the data in a separate window when applicable.

So, using the old and large PHP script, just to get it functioning, I had to bootstrap Drupal.
Place this at the beginning of OldLarge.php


$drupal_dir = getenv("DOCUMENT_ROOT");
$current_directory = getcwd();
chdir($drupal_dir);


require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

...
Do my Drupal stuff here....db_query() and whatever so that I can get results back from weird tables.
...

Then at the end of  OldLarge.php , add the final code....


chdir($current_directory);

Eventually...some day (which ends up being never) I will convert the OldLarge.php script to something more Drupalized.

CiviCRM "Javascript must be enabled in your browser in order to use the dashboard features"

"Javascript must be enabled in your browser in order to use the dashboard features"

I started getting this error from some of my testers when they were using IE9.  Sure enough, I couldn't access the CiviCRM dashboard or CiviCRM menu (it acted like it was disabled).

Yep, well JavaScript is enabled.

Stats on the site:
CiviCRM version 3.4.8
Latest Drupal (6.24)
Drupal module jquery_update installed (giving Drupal the jQuery version 1.3.2)

Hmmmmmm.....It works FINE with Chrome (16.0.912.77 m), Opera (11.61), Safari (5.1.2), and Firefox (10).

Oh YEAH!  That's right!!!   Internet Explorer doesn't handle a lot of JavaScript files very well.  CSS files are limited to 32(?) I think.  Not sure what the JS limit is.

So on the TEST site I enabled "Optimize JavaScript files" and all works well in IE8/IE9.

Things to look at:
http://drupal.org/project/javascript_aggregator
Great article by Vlad http://shvetsgroup.com/blog/optimizing-javascript-and-css-files-drupal

User Agents

A large list of user agents
http://techpatterns.com/forums/about304.html

Any other lists you know of?

My current sites/all/modules

My current "modules" folder and what it contains:

• civicrm
• contrib
• custom
• shortsitename_features

I kind of like that.  Nice and sweet.  There's a bunch of modules under "contrib" of course, a few under custom (but growing) and a few under shortsitename_features  (but growing).

CiviCRM - moving your configuration changes to a different environment

I love code driven development.  So when I build my Drupal sites, I immediately download and enable Features.  I do not want to handle website settings between environments by comparing screens and clicking box after box.  When you are running in a local DEV environment, plus you have a TEST and LIVE environment, using Features saves time (which of course equals money).

So I started a new project using CiviCRM and the thought in the back of my mind for awhile was:

"I just changed a CiviCRM setting.  How do I migrate that change to my TEST and LIVE environments?".  

Well the time came and I had to do it.  I needed to get my TEST site going and didn't want to bother remembering all of the settings I needed to configure.

Luckily, CiviCRM comes with migrate scripts to Export and Import your settings.

First Export it from the localhost (DEV environment) using one of the options.  More exist, but these are pretty basic:

1. Browse to
1.  http://localhost/virtual_dir/sites/all/modules/civicrm/bin/migrate/export.php?name=&pass=&key=
1. This will download a file called CustomGroupData.xml
2. Using wget (if you want to script this and update the file at 3pm for example)
1. wget -O D:\dev\htdocs\virtual_dir\sites\default\CiviCRM.info "http://localhost/virtual_dir/sites/all/modules/civicrm/bin/migrate/export.php?name=&pass=&key="

I save this file to a folder in the website I am working on.  Then, if it has changed, I commit it to git.

After the export file has been committed to git, then moved to your next server, import:
http://example.com/sites/all/modules/civicrm/bin/migrate/import.php?name=&pass=&key=&file=../../../../../default/CiviCRM.info


Notice the difference between this command and previous ones is that it uses import.php and specifies a file.

I noticed there were a couple of settings that did not get exported and I think those are related to CiviCRM Profiles.  I need to look into this a bit and see if this is an issue or "works as designed".

What are some other options for moving CiviCRM configuration changes between environments?

Drupal - Code Driven Development

I think most of my sites have 2 versions...LOCAL and LIVE to allow testing of various changes as well as developing new themes and modules. Some sites have 3 versions, LOCAL, USER TEST, and LIVE.

I like having a local version so that I can polish it first. Also, I use tools like NetBeans for coding and debugging.

I use "Code driven development" for all my Drupal sites. Remember, the changes you are making through the GUI in your Drupal site are being saved to the database.  To use code-driven development, you need to get those changes out of the DB and into code.  This is where Features comes in.  My process:

1. Develop code or theme; make your config changes in Drupal
2. Export as much as possible using Features and Strongarm - This puts the DB changes into code
3. Checkin the code to your version control system
4. On the server - pull code from your repo
5. Enable the various Features

Once you have your systems and procedures in place, this can actually be a time saver.  It's actually quite fun!!!! :)

Debugging PHP in NetBeans (in Ubuntu)

Here are the steps I used to setup PHP debugging in NetBeans (6.8 at the time of this writing) in Ubuntu 10.

sudo apt-get install php-pear
sudo apt-get install php5-dev
sudo apt-get install php5-xdebug



Edit your PHP INI settings
sudo nano /etc/php5/apache2/php.ini

Add the following lines under extensions:

zend_extension=/usr/lib/php5/the-folder/xdebug.so
xdebug.remote_enable=on

In my case it was ""/usr/lib/php5/20090626+lfs".  You will have to find the xdebug.so file in the correct folder.

Restart Apache
sudo /etc/init.d/apache2 restart


NetBeans PHP Debug Settings

1. Goto tools->Options
2. Select PHP
3. On the General tab
   
4. If this is your first time configuring xdebug, you may want to keep "Stop at First Line" checked.  this way when you start debugging you should know right away that your setup is correct. 

Start Debugging

• Add some break points.
• Right-click your project and select Debug

CentOS 5.6 server setup

These are the guides and blog posts I use to setup LAMP on my new CentOS servers:

• Get a newer version of PHP
• http://wiki.centos.org/HowTos/PHP_5.1_To_5.2
• Setup RPM Key
• cd /etc/pki/rpm-gpg
• wget http://download.fedora.redhat.com/pub/epel/RPM-GPG-KEY-EPEL
• Setup YUM repo for Git
• http://www.davegardner.me.uk/blog/2010/01/29/setting-up-git-on-centos-5-server/#install
• Install Apache, MySQL, PHP
• http://library.linode.com/lamp-guides/centos-5

I typically install Webmin too, so I'll document that one in a bit.