Browse Source

Initial commit

Sven Slootweg 7 years ago
commit
46adf150e5

+ 71 - 0
README.md

@ -0,0 +1,71 @@
1
## Disclaimer
2
3
All this code is very much hacked together to try and get things working in a reasonably reliable manner. It may have flaws, may be unpolished, or not even fully automated. Since pretty much noone uses HyperVM anymore anyway, I don't think this is too much of an issue.
4
5
**Do not under any circumstances attempt to use this code *without* first backing up all user and VPS data.** Use at your own risk.
6
7
This is going to take some attention and time to set up properly, and you'll be on your own for parts of it... but honestly, there's not really an alternative for mass HyperVM transfers.
8
9
## Caveats
10
11
* HyperVM is terrible. It may arbitrarily fail, and you may never find out why or how. Prepare to watch your migration status at regular intervals, taking over any transfers manually where necessary.
12
* **You cannot skip transfers.** While messing around with the database may give a similar result, it's not recommended. When a transfer fails, complete it manually and mark it as done manually (using `mark_done.py`).
13
* **Do not under any circumstances manually transfer a VPS to a new node during the migration process.** This will mess up the state in the migration database. Wait with manual migrations until the automated migrations have completed.
14
* To prevent issues, the migration towards a node will be suspended entirely when a failure happens. It will only resume after you have indicated that the transfer has been completed manually. Other nodes will continue migrating in the meantime.
15
* This usage guide was written afterwards. Doing this migration was largely a matter of trial and error, and I might be missing steps here. If you're stuck, you can always send an e-mail to admin@cryto.net, but I might simply not remember how I did something.
16
* **You should use the `admin` HyperVM user, and *not* an auxiliary account.**
17
* Reseller VPSes will fail to transfer and require manual transfer. The cause is apparently that a reseller VPS transfer has to be initiated from a different page, but I have not had the time to try and fix this.
18
19
## Requirements
20
21
* HyperVM.
22
* A HTTPd + PHP + MySQL setup.
23
* The Python `oursql` module.
24
25
## Data you need
26
27
1. A CSV database dump in the format `username,vpsid,currentnode,plan`.
28
2. A CSV database dump in the format `vpsid,email`.
29
3. A CSV list of VPS sizes in the format `size,vpsid`. You can gather this through running `du -sh /vz/private/*` on each node, creating one giant list out of results from all nodes, and replacing tabs with commas.
30
31
## Setup instructions for environment
32
33
1. Set up a new VPS (that will not be transfered!) with the environment as listed in the Requirements section.
34
2. Create a new database, and import `structure.sql` into it.
35
3. Clone the repository, and be sure to move everything in `public_html` to your document root for the HTTPd.
36
4. Edit the `.php` files in the document root to set the database configuration.
37
38
## Setup instructions for migration schedule
39
40
1. Create a file named `input.csv` with the data from the first database dump.
41
2. Modify sort.py. The `locations` variable holds a structure with all the nodes for the current locations. `new_servers` holds a structure defining how many new nodes exist for each 'category' in each location. In the existing script, these are marked as 12, 34, and 56, but you can use any numeric value here. If you increase or decrease the amount of categories, be sure to change the code elsewhere to reflect this. Change the code from line 43 to 48, to sort the plans into the right categories. Change the hostname in line 73 to reflect your hostname format.
42
3. Run sort.py, redirecting stdout to a file named `sorted.csv`. You now have your existing VPSes planned uniformly across the number of nodes you have for each category in each location.
43
4. Edit `schedule.py`, `emails.py`, `reorder_size.py`, `failed.py`, `status.py`, `mark_done.py`, and `run.py`, and set the correct database configuration in each.
44
5. Run `schedule.py`.
45
6. Create a new file containing the second database dump as `emails.csv`.
46
7. Run `emails.py`.
47
8. Create a new file containing the third database dump as `sizes.csv`.
48
9. Run `reorder_size.py`.
49
50
## Other setup stuff
51
52
1. Edit `run.py` to use the correct SMTP login details, sender address, and panel login details.
53
2. Edit the e-mail templates in `run.py` to reflect what you want to send to your users.
54
3. Edit `report.php` to set a 'reporting key' that is used by the HyperVM panel to authenticate itself to your reporting script.
55
4. Back up the original `livemigrate.php` and `switchserver.php` in `hypervm/bin/common/` (you'll need to restore these after the migration!).
56
5. Copy the patched files from the `patch` directory into the `hypervm/bin/common` directory.
57
6. Edit the reporting key in the patched `livemigrate.php` and `switchserver.php` files. Note that it has to be replaced in two places in each of the files!
58
7. Edit the hostname that the patched migration scripts should report to.
59
60
## Running the migration
61
62
* The migration process is largely automatic. You do need to pay regular attention to the status to resolve any issues.
63
* `run.py` will start the migration. Using the `--resume` flag will make it skip the initial notification e-mail phase. Do this when for whatever reason the script has terminated, and you wish to restart it.
64
* If a transfer fails (or gets stuck in the 'busy' phase), ensure that it has finished transfering and manually transfer it if required. Afterwards, run `mark_done.py` with the name of the target node as first parameter. It will assume that the currently transfering/failed VPS has finished migrating, send a success e-mail, and move on to the next.
65
* `status.py` will give you a human-readable overview of the current status for each node.
66
* `failed.py` will give you an overview of the relevant information for all transfers that are in the 'failed' state, and need manual attention.
67
* All sent e-mails are logged to `email.txt`.
68
69
## License
70
71
All this, except for obviously the BlueVM logo, is licensed under the [WTFPL](http://wtfpl.net/). Reuse as you wish. [Donations are very much appreciated](http://cryto.net/~joepie91/donate.html).

+ 121 - 0
patch/livemigrate.php

@ -0,0 +1,121 @@
1
<?php 
2
3
include_once "htmllib/lib/include.php";
4
5
switchserver_main();
6
7
function switchserver_main()
8
{
9
10
	global $argc, $argv;
11
	global $gbl, $sgbl, $login, $ghtml; 
12
13
	//sleep(60);
14
	initProgram("admin");
15
16
	if ($argc === 1) {
17
		print("Usage: $argv[0] --class= --name= --v-syncserver= \n");
18
		exit;
19
	}
20
21
22
	try {
23
		$opt = parse_opt($argv);
24
25
		$param = get_variable($opt);
26
27
		dprintr($param);
28
29
		$class = $opt['class'];
30
		$name = $opt['name'];
31
32
		if (lx_core_lock("$class-$name.livemigrate")) {
33
			exit;
34
		}
35
36
		$object = new $class(null, 'localhost', $name);
37
		$object->get();
38
		if ($object->dbaction === 'add') {
39
			throw new lxException ("no_object", '', '');
40
			exit;
41
		}
42
43
		if (!$object->syncserver) {
44
			print("No_synserver...\n");
45
			throw new lxException ("no_syncserver", '', '');
46
			exit;
47
		}
48
49
		if ($param['syncserver'] === $object->syncserver) {
50
			print("No Change...\n");
51
			throw new lxException ("no_change", '', '');
52
			exit;
53
		}
54
55
56
57
		$driverapp_old = $gbl->getSyncClass('localhost', $object->syncserver, $object->getClass());
58
		$driverapp_new = $gbl->getSyncClass('localhost', $param['syncserver'], $object->getClass());
59
60
		$oldserver = $object->syncserver;
61
		$newserver = $param['syncserver'];
62
63
		if ($driverapp_new !== $driverapp_old) {
64
			throw new lxException ("the_drivers_are_different_in_two_servers", '', '');
65
		}
66
67
		$actualserver = getFQDNforServer($newserver);
68
69
		setup_ssh_channel($oldserver, $newserver, $actualserver);
70
71
72
		$ssh_port = db_get_value("sshconfig", $newserver, "ssh_port");
73
		if (!$ssh_port) { $ssh_port = "22" ; }
74
75
		$res = rl_exec_get(null, $oldserver, "exec_vzmigrate", array($object->vpsid, $actualserver, $ssh_port));
76
77
		list($ret, $error)  = $res;
78
79
80
		if ($ret !== 0) {
81
			throw new lxException ("vzmigrate_failed_due_to:$error", '', '');
82
		}
83
84
		$object->olddeleteflag = 'done';
85
		$object->syncserver = $newserver;
86
87
		$object->username = null;
88
		$object->makeSureTheUserExists();
89
		$object->setUpdateSubaction();
90
		$object->write();
91
92
		
93
	} catch (exception $e) {
94
		print($e->getMessage());
95
		/// hcak ahck... Chnage only the olddelete variable which is the mutex used for locking in the process of switch. The problem is we want to totally bail out if the switchserver fails. The corect way would be save after reverting the syncserve to the old value, but that's a bit risky. So we just use a hack to change only the olddeleteflag; Not a real hack.. This is the better way.
96
97
		$message = "{$e->getMessage()}";
98
99
100
		$message = str_replace("'", "", $message);
101
		write_to_object($object, $message, $param['syncserver']);
102
		$fullmesage = "Switch of {$object->getClass()}:{$object->nname} to {$param['syncserver']} failed due to {$e->getMessage()}";
103
		file_get_contents("http://transfer.bluevm.com/report.php?action=failed&key=keygoeshere&server={$param['syncserver']}");
104
		log_switch($fullmesage);
105
106
		mail($login->contactemail, "Switch Failed:", "$fullmesage\n");
107
		print("\n");
108
		exit;
109
	}
110
	mail($login->contactemail, "Switch Succeeded", "Switch Succeeded {$object->getClass()}:$object->nname to {$param['syncserver']}\n");
111
	file_get_contents("http://transfer.bluevm.com/report.php?action=done&key=keygoeshere&server={$param['syncserver']}");
112
}
113
114
115
116
function write_to_object($object, $message, $syncserver)
117
{
118
	$sq = new Sqlite(null, $object->__table);
119
	$message = str_replace("'", "", $message);
120
	$sq->rawQuery("update {$object->__table} set olddeleteflag = 'Switch to $syncserver failed due to $message' where nname = '{$object->nname}'");
121
}

+ 93 - 0
patch/switchserver.php

@ -0,0 +1,93 @@
1
<?php 
2
3
include_once "htmllib/lib/include.php";
4
5
switchserver_main();
6
7
function switchserver_main()
8
{
9
10
	global $argc, $argv;
11
	global $gbl, $sgbl, $login, $ghtml; 
12
13
	//sleep(60);
14
	initProgram("admin");
15
16
	if ($argc === 1) {
17
		print("Usage: $argv[0] --class= --name= --v-syncserver= \n");
18
		exit;
19
	}
20
21
22
	try {
23
		$opt = parse_opt($argv);
24
25
		$param = get_variable($opt);
26
27
		dprintr($param);
28
29
		$class = $opt['class'];
30
		$name = $opt['name'];
31
32
		if (lx_core_lock("$class-$name.switchserver")) {
33
			exit;
34
		}
35
36
		$object = new $class(null, 'localhost', $name);
37
		$object->get();
38
		if ($object->dbaction === 'add') {
39
			throw new lxException ("no_object", '', '');
40
			exit;
41
		}
42
43
		if (!$object->syncserver) {
44
			print("No_synserver...\n");
45
			throw new lxException ("no_syncserver", '', '');
46
			exit;
47
		}
48
49
		if ($param['syncserver'] === $object->syncserver) {
50
			print("No Change...\n");
51
			throw new lxException ("no_change", '', '');
52
			exit;
53
		}
54
55
56
57
		$driverapp_old = $gbl->getSyncClass('localhost', $object->syncserver, $object->get__table());
58
		$driverapp_new = $gbl->getSyncClass('localhost', $param['syncserver'], $object->get__table());
59
60
		if ($driverapp_new !== $driverapp_old) {
61
			//throw new lxException ("the_drivers_are_different_in_two_servers", '', '');
62
		}
63
64
65
		$object->doupdateSwitchserver($param);
66
	} catch (exception $e) {
67
		print($e->getMessage());
68
		/// hcak ahck... Chnage only the olddelete variable which is the mutex used for locking in the process of switch. The problem is we want to totally bail out if the switchserver fails. The corect way would be save after reverting the syncserve to the old value, but that's a bit risky. So we just use a hack to change only the olddeleteflag; Not a real hack.. This is the better way.
69
70
		$message = "{$e->getMessage()}";
71
72
73
		write_to_object($object, $message, $param['syncserver']);
74
		$fullmesage = "Switch of {$object->get__table()}:{$object->nname} to $object->syncserver failed due to {$e->getMessage()}";
75
		file_get_contents("http://transfer.bluevm.com/report.php?action=failed&key=keygoeshere&server={$param['syncserver']}");
76
		log_switch($fullmesage);
77
78
		mail($login->contactemail, "Switch Failed:", "$fullmesage\n");
79
		print("\n");
80
		exit;
81
	}
82
	mail($login->contactemail, "Switch Succeeded", "Switch Succeeded {$object->get__table()}:$object->nname to {$param['syncserver']}\n");
83
	file_get_contents("http://transfer.bluevm.com/report.php?action=done&key=keygoeshere&server={$param['syncserver']}");
84
	
85
}
86
87
88
89
function write_to_object($object, $message, $syncserver)
90
{
91
	$sq = new Sqlite(null, $object->get__table());
92
	$sq->rawQuery("update {$object->get__table()} set olddeleteflag = 'Switch to $syncserver failed due to $message' where nname = '{$object->nname}'");
93
}

+ 120 - 0
public_html/index.php

@ -0,0 +1,120 @@
1
<?php
2
$database = new PDO("mysql:host=localhost;dbname=transfer", "root", "");
3
4
if(empty($_GET['email']) || empty($_GET['key']))
5
{
6
	die("No valid email and key specified. Please check the email you received about the migration of your servers.");
7
}
8
9
$statement = $database->prepare("SELECT * FROM emails WHERE `EmailAddress` = ? AND `Key` = ?");
10
$statement->bindValue(1, $_GET['email'], PDO::PARAM_STR);
11
$statement->bindValue(2, $_GET['key'], PDO::PARAM_STR);
12
$statement->execute();
13
$results = $statement->fetchAll();
14
15
if(count($results) == 0)
16
{
17
	die("No valid email and key specified. Please check the email you received about the migration of your servers.");
18
}
19
20
$statement = $database->prepare("SELECT * FROM entries WHERE `EmailAddress` = ? ORDER BY `Finished` DESC");
21
$statement->bindValue(1, $_GET['email'], PDO::PARAM_STR);
22
$statement->execute();
23
$vpses = $statement->fetchAll();
24
25
?>
26
27
<!doctype html>
28
<html>
29
	<head>
30
		<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
31
		<link rel="stylesheet" href="style.css">
32
		<link href='http://fonts.googleapis.com/css?family=Orienta' rel='stylesheet' type='text/css'>
33
	</head>
34
	<body>
35
		<div class="header">
36
			<img src="logo.png">
37
		</div>
38
		<div class="main">
39
			<h1>VPS migration status</h1>
40
			<table>
41
				<tr>
42
					<th></th>
43
					<th>Username</th>
44
					<th>Status</th>
45
				</tr>
46
				<?php
47
				foreach($vpses as $row)
48
				{
49
					?>
50
					<tr>
51
						<?php if($row['Finished'] == "0") 
52
						{ 
53
							?>
54
							<td class="status"><span class="done">A</span> ⇒ B</td>
55
							<?php
56
						}
57
						elseif($row['Finished'] == "1")
58
						{
59
							?>
60
							<td class="status">A <span class="done"><img src="processing.gif"></span> B</td>
61
							<?php
62
						}
63
						elseif($row['Finished'] == "2")
64
						{
65
							?>
66
							<td class="status">A ⇒ <span class="done">B</span></td>
67
							<?php
68
						}
69
						elseif($row['Finished'] == "3")
70
						{
71
							?>
72
							<td class="status">A ⇒ B</td>
73
							<?php
74
						}
75
						?>
76
						
77
						<td><?php echo(htmlspecialchars($row["Username"])); ?></td>
78
						
79
						<?php if($row['Finished'] == "0") 
80
						{
81
							$my_pos = $row['Position'];
82
							
83
							$statement = $database->prepare("SELECT * FROM servers WHERE `Host` = ?");
84
							$statement->bindValue(1, $row['TargetNode'], PDO::PARAM_STR);
85
							$statement->execute();
86
							$node = $statement->fetch();
87
							$current_pos = $node['Current'];
88
							
89
							$real_pos = $my_pos - $current_pos;
90
							?>
91
							<td class="queue">Queued in position <?php echo($real_pos); ?>.</td>
92
							<?php
93
						}
94
						elseif($row['Finished'] == "1")
95
						{
96
							?>
97
							<td class="queue">Transferring...</td>
98
							<?php
99
						}
100
						elseif($row['Finished'] == "2")
101
						{
102
							?>
103
							<td class="queue">Done!</td>
104
							<?php
105
						}
106
						elseif($row['Finished'] == "3")
107
						{
108
							?>
109
							<td class="queue">Failed</td>
110
							<?php
111
						}
112
						?>
113
					</tr>
114
					<?php
115
				}	
116
				?>
117
			</table>
118
		</div>
119
	</body>
120
</html>

BIN
public_html/logo.png


BIN
public_html/processing.gif


+ 28 - 0
public_html/report.php

@ -0,0 +1,28 @@
1
<?php
2
$db = "schedule.db";
3
4
$database = new PDO("mysql:host=localhost;dbname=transfer", "root", "");
5
6
$database->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
7
8
if($_GET['key'] == "keygoeshere")
9
{
10
	if($_GET['action'] == "done")
11
	{
12
		$statement = $database->prepare("UPDATE entries SET `Finished` = 2 WHERE `Finished` = 1 AND `TargetNode` = ?");
13
		$statement->bindValue(1, $_GET['server'], PDO::PARAM_STR);
14
		$statement->execute();
15
		$statement = $database->prepare("UPDATE servers SET `Busy` = 0 WHERE `Host` = ? AND `Busy` != 4");
16
		$statement->bindValue(1, $_GET['server'], PDO::PARAM_STR);
17
		$statement->execute();
18
	}
19
	elseif($_GET['action'] == "failed")
20
	{
21
		$statement = $database->prepare("UPDATE entries SET `Finished` = 3 WHERE `Finished` = 1 AND `TargetNode` = ?");
22
		$statement->bindValue(1, $_GET['server'], PDO::PARAM_STR);
23
		$statement->execute();
24
		$statement = $database->prepare("UPDATE servers SET `Busy` = 2 WHERE `Host` = ? AND `Busy` != 4");
25
		$statement->bindValue(1, $_GET['server'], PDO::PARAM_STR);
26
		$statement->execute();
27
	}
28
}

+ 89 - 0
public_html/style.css

@ -0,0 +1,89 @@
1
body
2
{
3
	font-family: Orienta, sans-serif;
4
	margin: 0px;
5
	padding: 0px;
6
}
7
8
.header
9
{
10
	background-color: #93B5FB;
11
	border-bottom: 1px solid #5D97FF;
12
	padding: 12px;
13
}
14
15
.main
16
{
17
	padding: 12px;
18
}
19
20
h1
21
{
22
	margin-top: 4px;
23
}
24
25
table
26
{
27
	border-spacing: 0px;
28
	border-radius: 11px;
29
}
30
31
th, td
32
{
33
	padding: 9px 12px;
34
	border-top: 1px solid #929292;
35
	border-right: 1px solid #929292;
36
	text-align: left;
37
}
38
39
th:first-child 
40
{
41
	border-radius: 6px 0 0 0;
42
}
43
44
th:last-child 
45
{
46
	border-radius: 0 6px 0 0;
47
}
48
49
tr:last-child td:first-child
50
{
51
	border-radius: 0 0 0 6px;
52
}
53
54
tr:last-child td:last-child
55
{
56
	border-radius: 0 0 6px 0;
57
}
58
59
tr:last-child td
60
{
61
	border-bottom: 1px solid #929292;
62
}
63
64
td:first-child, th:first-child
65
{
66
	border-left: 1px solid #929292;
67
}
68
69
th:only-child
70
{
71
	border-radius: 6px 6px 0 0;
72
}
73
74
td.status
75
{
76
	color: silver;
77
	font-size: 18px;
78
	font-weight: bold;
79
}
80
81
td.status .done
82
{
83
	color: #128310;
84
}
85
86
td.queue
87
{
88
	
89
}

+ 26 - 0
script/emails.py

@ -0,0 +1,26 @@
1
import oursql, csv, random, string
2
3
# input.csv must be in the format vpsid,email
4
5
db = oursql.connect(host="localhost", user="root", passwd="", db="transfer")
6
reader = csv.reader(open("emails.csv", "r"))
7
8
emails = []
9
c = db.cursor()
10
11
for entry in reader:
12
	vps_id, email = entry
13
	c.execute("UPDATE entries SET `EmailAddress` = ? WHERE `VpsId` = ?", (email, vps_id))
14
	print "Added e-mail address for VPS %s" % (vps_id)
15
	
16
	if email not in emails:
17
		emails.append(email)
18
19
db.commit()
20
21
for email in emails:
22
	if email.strip() != "":
23
		random_key = ''.join(random.choice(string.ascii_uppercase + string.digits) for x in xrange(0, 16))
24
		c.execute("INSERT INTO emails (`EmailAddress`, `Key`) VALUES (?, ?)", (email, random_key))
25
26
db.commit()

+ 23 - 0
script/failed.py

@ -0,0 +1,23 @@
1
#!/usr/bin/python
2
3
import oursql
4
5
db = oursql.connect(host="localhost", user="root", passwd="", db="transfer")
6
c = db.cursor()
7
8
c.execute("SELECT * FROM servers")
9
servers = c.fetchall()
10
11
for server in servers:
12
	if server[2] == 2 or server[2] == 4:
13
		c.execute("SELECT * FROM entries WHERE `TargetNode` = ? AND `Position` = ?", (server[1], server[3]))
14
		results = c.fetchall()
15
		
16
		try:
17
			vps = results[0]
18
		except IndexError, e:
19
			continue
20
		
21
		if vps[6] == 3:
22
			print "%s\t%s\t%s\t%s" % (server[1].ljust(26), str(vps[1]).ljust(6), vps[2].ljust(18), vps[3])
23
	

+ 12 - 0
script/mark_done.py

@ -0,0 +1,12 @@
1
#!/usr/bin/python
2
3
import oursql, sys
4
5
db = oursql.connect(host="localhost", user="root", passwd="", db="transfer")
6
c = db.cursor()
7
8
server = sys.argv[1]
9
10
c.execute("UPDATE servers SET `Busy` = 0 WHERE `Host` = '%s'" % server)
11
c.execute("UPDATE entries SET `Finished` = 2 WHERE (`Finished` = 3 OR `Finished` = 1) AND `TargetNode` = '%s'" % server)
12

+ 37 - 0
script/reorder_size.py

@ -0,0 +1,37 @@
1
import csv, re, operator, oursql
2
3
db = oursql.connect(host="localhost", user="root", passwd="", db="transfer")
4
reader = csv.reader(open("sizes.csv", "r"))
5
6
sizes = {}
7
8
for entry in reader:
9
	size, vpsid = entry
10
	
11
	if re.match("[0-9]+", vpsid):
12
		sizes[vpsid] = int(size)
13
		#print "%s => %.2fG" % (vpsid, int(size) / 1024.0 / 1024.0)
14
15
sorted_list = sorted(sizes.iteritems(), key=operator.itemgetter(1))
16
17
c = db.cursor()
18
19
for vpsid, size in sorted_list:
20
	c.execute("UPDATE entries SET `DiskUsage` = ? WHERE `VpsID` = ?", (size, vpsid))
21
	
22
c.execute("SELECT * FROM entries ORDER BY `TargetNode` ASC, `DiskUsage` ASC")
23
results = c.fetchall()
24
25
servers = {}
26
27
for row in results:
28
	nodename = row[4]
29
	
30
	try:
31
		servers[nodename] += 1
32
	except KeyError, e:
33
		servers[nodename] = 1
34
		
35
	c.execute("UPDATE entries SET `Position` = ? WHERE `Id` = ?", (servers[nodename], row[0]))
36
37

+ 369 - 0
script/run.py

@ -0,0 +1,369 @@
1
#!/usr/bin/env python
2
3
import oursql, time, sys, requests, re
4
import smtplib
5
import datetime
6
7
db = oursql.connect(host="localhost", user="root", passwd="", db="transfer")
8
9
mail_start = """
10
Hello,
11
12
As announced a short while ago, we are going to migrate your VPSes to new servers.
13
We've had some unforeseen technical difficulties that led to a delay, but these
14
have been resolved. This e-mail is to inform you that the migration process has 
15
been started.
16
17
The following of your VPSes are affected:
18
19
%s
20
21
You can follow the queue position and progress of your VPS migrations here: %s
22
23
You will receive further notifications for every individual VPS that is being migrated.
24
Please ensure that you add admin@bluevm.com to your address book, so that
25
our e-mails do not end up in your spam folder.
26
27
Regards,
28
BlueVM Staff
29
https://bluevm.com/
30
"""
31
32
mail_vps_start = """
33
Hello,
34
35
This e-mail is to inform you that the migration of your VPS with the username %s has started.
36
You will receive an e-mail when this migration is finished.
37
38
You can follow the progress of all your VPS migrations at %s
39
40
Regards,
41
BlueVM Staff
42
https://bluevm.com/
43
"""
44
45
mail_vps_success = """
46
Hello,
47
48
This e-mail is to inform you that the migration of your VPS with the username %s has 
49
finished successfully.
50
51
Your new IP addresses are:
52
53
%s
54
55
You can follow the progress of all your VPS migrations at %s
56
57
Regards,
58
BlueVM Staff
59
https://bluevm.com/
60
"""
61
62
mail_vps_error = """
63
Hello,
64
65
This e-mail is to inform you that the migration of your VPS with the username %s has 
66
been aborted because an error has occurred. No data loss has occurred, and staff
67
will be looking into the issue shortly.
68
69
You can follow the progress of all your VPS migrations at %s
70
71
Regards,
72
BlueVM Staff
73
https://bluevm.com/
74
"""
75
76
class Panel(object):
77
	def __init__(self, target):
78
		self.target = target
79
		self.sess = requests.Session()
80
		
81
	def login(self, username, password):
82
		page = self.sess.get("%s/login/" % self.target).text
83
		match = re.search('<input type=hidden name=id value="([^"]+)">', page)
84
85
		if match is None:
86
			raise Exception("Not a valid HyperVM panel.")
87
			
88
		fid = match.group(1)
89
90
		error_string = "Login Unsuccessful"
91
		post_target = "%s/htmllib/phplib/" % self.target
92
		post_vars = {
93
			"frm_clientname": username,
94
			"frm_password": password,
95
			"id": fid,
96
			"login": "Login"
97
		}
98
99
		page = self.sess.post(post_target, data=post_vars).text
100
101
		if error_string in page:
102
			raise Exception("Login failed.")
103
		
104
	def run_command(self, node, command):
105
		post_target = "%s/display.php" % self.target
106
		post_vars = {
107
			"frm_o_o[0][class]": "pserver",
108
			"frm_o_o[0][nname]": node,
109
			"frm_pserver_c_ccenter_command": command,
110
			"frm_pserver_c_ccenter_error": "",
111
			"frm_action": "updateform",
112
			"frm_subaction": "commandcenter",
113
			"frm_change": "Execute"
114
		}
115
116
		page = self.sess.post(post_target, data=post_vars).text
117
		result = re.search("<textarea[^>]+frmtextarea[^>]+>(.*?)<\/textarea>", page, re.DOTALL)
118
		
119
		if result is not None:
120
			return result.group(1)
121
		
122
	def Vps(self, username):
123
		return Vps(self, username)
124
	
125
class Vps(object):
126
	def __init__(self, panel, username):
127
		self.panel = panel
128
		self.username = username
129
		
130
	def get_ip_addresses(self):
131
		get_target = "%s/display.php" % self.panel.target
132
		get_vars = {
133
			"frm_o_o[0][class]": "vps",
134
			"frm_o_o[0][nname]": self.username,
135
			"frm_o_cname": "vmipaddress_a",
136
			"frm_action": "list",
137
			"frm_list_refresh": "yes"
138
		}
139
140
		page = self.panel.sess.get(get_target, params=get_vars).text
141
142
		ip_regex = '<td\s+class=collist\s+wrap\s+align=left\s+>\s+([0-9.]+)\s+<\/td>'
143
		ip_list = re.findall(ip_regex, page)
144
		self.ip_list = ip_list
145
		
146
		return ip_list
147
		
148
	def delete_ip_addresses(self, addresses):
149
		post_target = "%s/display.php" % self.panel.target
150
		post_vars = {
151
			"frm_o_o[0][class]": "vps",
152
			"frm_o_o[0][nname]": self.username,
153
			"frm_accountselect": ",".join(addresses),
154
			"frm_action": "delete",
155
			"frm_o_cname": "vmipaddress_a",
156
			"frm_confirmed": "yes",
157
			"Confrm": "Confirm"
158
		}
159
160
		page = self.panel.sess.post(post_target, data=post_vars).text
161
		
162
	def add_ip_addresses_num(self, num):
163
		post_target = "%s/display.php" % self.panel.target
164
		post_vars = {
165
			"frm_o_o[0][class]": "vps",
166
			"frm_o_o[0][nname]": self.username,
167
			"frm_dttype[var]": "type",
168
			"frm_dttype[val]": "npool",
169
			"frm_o_cname": "vmipaddress_a",
170
			"frm_vmipaddress_a_c_type": "npool",
171
			"frm_vmipaddress_a_c_ip_num": num,
172
			"frm_action": "add",
173
			"frm_change": "Add"
174
		}
175
		
176
		page = self.panel.sess.post(post_target, data=post_vars).text
177
		
178
	def transfer_to_live(self, target):
179
		post_target = "%s/display.php" % self.panel.target
180
		post_vars = {
181
			"frm_o_o[0][class]": "vps",
182
			"frm_o_o[0][nname]": self.username,
183
			"frm_vps_c_syncserver": target,
184
			"frm_action": "update",
185
			"frm_subaction": "livemigrate",
186
			"frm_change": "Update"
187
		}
188
189
		page = self.panel.sess.post(post_target, data=post_vars).text
190
191
		if "Switching the Servers has been run in the background." not in page:
192
			raise Exception("Transfer of %s to %s failed. Maybe an incorrect destination node was specified?" % (self.username, target))
193
			
194
	def transfer_to(self, target):
195
		post_target = "%s/display.php" % self.panel.target
196
		post_vars = {
197
			"frm_o_o[0][class]": "vps",
198
			"frm_o_o[0][nname]": self.username,
199
			"frm_vps_c_syncserver": target,
200
			"frm_action": "update",
201
			"frm_subaction": "switchserver",
202
			"frm_change": "Update"
203
		}
204
205
		page = self.panel.sess.post(post_target, data=post_vars).text
206
207
		if "Switching the Servers has been run in the background." not in page:
208
			raise Exception("Transfer of %s to %s failed. Maybe an incorrect destination node was specified?" % (self.username, target))
209
			
210
	def boot(self):
211
		get_target = "%s/display.php" % self.panel.target
212
		get_vars = {
213
			"frm_o_o[0][class]": "vps",
214
			"frm_o_o[0][nname]": self.username,
215
			"frm_action": "update",
216
			"frm_subaction": "boot"
217
		}
218
219
		page = self.panel.sess.get(get_target, params=get_vars).text
220
221
class Mailserver(object):
222
	def __init__(self, ip, port, ssl, user, password):
223
		if ssl == True:
224
			self.smtp = smtplib.SMTP_SSL()
225
		else:
226
			self.smtp = smtplib.SMTP()
227
			
228
		self.smtp.set_debuglevel(0)
229
		self.smtp.connect(ip, port)
230
		self.smtp.login(user, password)
231
		
232
	def send(self, to, subject, body):
233
		from_addr = "BlueVM <admin@bluevm.com>"
234
		date = datetime.datetime.now().strftime("%d/%m/%Y %H:%M")
235
		msg = "From: %s\nTo: %s\nSubject: %s\n\n%s\n" % (from_addr, to, subject, body)
236
		
237
		self.smtp.sendmail(from_addr, to, msg)
238
		
239
	def quit(self):
240
		self.smtp.quit()
241
242
def send_mail(to, subject, body):
243
	f = open("email.txt", "a")
244
	f.write("To: %s\nSubject: %s\n\n%s\n\n\n" % (to, subject, body))
245
	f.close()
246
	
247
	mailserv = Mailserver("smtp.sendgrid.net", 465, True, "user", "pass")
248
	mailserv.send(to, subject, body)
249
	mailserv.quit()
250
	
251
def generate_url(email):
252
	global db
253
	c = db.cursor()
254
	c.execute("SELECT * FROM emails WHERE `EmailAddress` = ?", (email,))
255
	result = c.fetchall()[0]
256
	return "http://transfer.bluevm.com/?email=%s&key=%s" % (result[1], result[2])
257
258
panel = Panel("http://manage.bluevm.com:8888")
259
panel.login("admin", "password")
260
print "Logged in."
261
262
c = db.cursor()
263
264
c.execute("SELECT * FROM emails")
265
emails = c.fetchall()
266
total = len(emails)
267
done = 0
268
269
try:
270
	resumed = (sys.argv[1] == "--resume")
271
except IndexError, e:
272
	resumed = False
273
274
if resumed == False:
275
	for email in emails:
276
		c.execute("SELECT * FROM entries WHERE `EmailAddress` = ?", (email[1],))
277
		entries = c.fetchall()
278
		
279
		if len(entries) > 0:
280
			send_mail(email[1], "Migration process started", mail_start % ("\n\n".join(entry[2] for entry in entries), generate_url(email[1])))
281
		
282
		done += 1
283
		
284
		sys.stdout.write("Sent %d of %d announcement emails.\n" % (done, total))
285
286
while True:
287
	c.execute("SELECT * FROM servers WHERE `Busy` = 2")
288
	servers = c.fetchall()
289
	
290
	for server in servers:
291
		c.execute("SELECT * FROM entries WHERE `TargetNode` = ? AND `Position` = ?", (server[1], server[3]))
292
		vpses = c.fetchall()
293
		
294
		c.execute("UPDATE servers SET `Busy` = 4 WHERE `Id` = ?", (server[0],))
295
		
296
		if len(vpses) > 0:
297
			send_mail(vpses[0][3], "VPS migration failed", mail_vps_error % (vpses[0][2], generate_url(vpses[0][3])))
298
			sys.stderr.write("WARNING: Transfer of %s to %s failed! Further transfers for this server have been aborted.\n" % (vpses[0][2], server[1]))
299
		else:
300
			sys.stderr.write("WARNING: Transfer of VPS to %s failed! Further transfers for this server have been aborted.\n" % (server[1]))
301
	
302
	c.execute("SELECT * FROM servers WHERE `Busy` = 0")
303
	servers = c.fetchall()
304
	
305
	if len(servers) > 0:
306
		for server in servers:
307
			c.execute("SELECT * FROM entries WHERE `TargetNode` = ? AND `Position` = ? LIMIT 1", (server[1], server[3]))
308
			results = c.fetchall()
309
			
310
			if len(results) > 0:
311
				last = results[0]
312
				
313
				vps = panel.Vps(last[2])
314
				ip_list = vps.get_ip_addresses()
315
				ip_count = len(ip_list)
316
317
				if ip_count > 0:
318
					vps.delete_ip_addresses(ip_list)
319
					sys.stderr.write("[%s] Deleted %d IP addresses: %s\n" % (last[2], ip_count, ", ".join(ip_list)))
320
321
					vps.add_ip_addresses_num(ip_count)
322
					sys.stderr.write("[%s] Added %d IP addresses from pool.\n" % (last[2], ip_count))
323
					
324
				result = panel.run_command(server[1], "mkdir /vz/root/%s" % last[1])
325
				
326
				try:
327
					if result.strip() == "":
328
						sys.stdout.write("[%s] Fixed root directory.\n" % last[2])
329
				except AttributeError, e:
330
					sys.stderr.write("[%s] WARNING: Could not fix root directory" % last[2])
331
					
332
				vps.boot()
333
				sys.stdout.write("[%s] Booted.\n" % last[2])
334
					
335
				send_mail(last[3], "VPS migration finished", mail_vps_success % (last[2], "\n\n".join(vps.get_ip_addresses()), generate_url(last[3])))
336
337
			
338
			c.execute("SELECT * FROM entries WHERE `TargetNode` = ? AND `Finished` = 0 ORDER BY `Position` ASC LIMIT 1", (server[1],))
339
			entries = c.fetchall()
340
			
341
			if len(entries) == 0:
342
				c.execute("UPDATE servers SET `Busy` = 3 WHERE `Id` = ?", (server[0],))
343
				sys.stdout.write("Server %s finished migrating.\n" % server[1])
344
			else:
345
				eid = entries[0][0]
346
				c.execute("UPDATE entries SET `Finished` = 1 WHERE `Id` = ?", (eid,))
347
				c.execute("UPDATE servers SET `Busy` = 1, `Current` = `Current` + 1 WHERE `Id` = ?", (server[0],))
348
				
349
				send_mail(entries[0][3], "VPS migration started", mail_vps_start % (entries[0][2], generate_url(entries[0][3])))
350
				sys.stdout.write("[%s] Started transfer to %s\n" % (entries[0][2], entries[0][4]))
351
				
352
				vps = panel.Vps(entries[0][2])
353
				
354
				try:
355
					vps.transfer_to(entries[0][4])
356
				except Exception, e:
357
					c.execute("UPDATE servers SET `Busy` = 4 WHERE `Host` = ?", (entries[0][4],))
358
					c.execute("UPDATE entries SET `Finished` = 3 WHERE `Id` = ?", (entries[0][0],))
359
					send_mail(entries[0][3], "VPS migration failed", mail_vps_error % (entries[0][2], generate_url(entries[0][3])))
360
					sys.stderr.write("[%s] WARNING: Transfer to %s failed! Further transfers for this server have been aborted.\n" % (entries[0][2], entries[0][4]))
361
	else:
362
		c.execute("SELECT * FROM servers WHERE `Busy` != 3 AND `Busy` != 4")
363
		
364
		if len(c.fetchall()) == 0:
365
			sys.stdout.write("All migrations done. Check the log for any errors.\n")
366
			exit(0)
367
	
368
		
369
	time.sleep(5)

+ 32 - 0
script/schedule.py

@ -0,0 +1,32 @@
1
import oursql, csv
2
3
# input.csv must be in the format username,vpsid,targetnode
4
5
db = oursql.connect(host="localhost", user="root", passwd="", db="transfer")
6
reader = csv.reader(open("sorted.csv", "r"))
7
8
nodes = []
9
c = db.cursor()
10
11
for entry in reader:
12
	username, vps_id, target_node = entry
13
	c.execute("INSERT INTO entries (`VpsId`, `Username`, `TargetNode`, `Finished`) VALUES (?, ?, ?, 0)", (vps_id, username, target_node))
14
	print "Adding VPS %s to queue (target node %s)" % (vps_id, target_node)
15
	
16
	if target_node not in nodes:
17
		nodes.append(target_node)
18
19
db.commit()
20
21
for node in nodes:
22
	c.execute("INSERT INTO servers (`Host`, `Busy`, `Current`) VALUES (?, 0, 0)", (node,))
23
	
24
	c.execute("SELECT Id FROM entries WHERE `TargetNode` = ?", (node,))
25
	node_entries = c.fetchall()
26
	counter = 1
27
	
28
	for entry in node_entries:
29
		c.execute("UPDATE entries SET `Position` = ? WHERE `Id` = ?", (counter, entry[0]))
30
		counter += 1
31
32
db.commit()

+ 76 - 0
script/sort.py

@ -0,0 +1,76 @@
1
#!/usr/bin/env python
2
from collections import OrderedDict
3
import csv
4
5
reader = csv.reader(open("input.csv", "r"))
6
sorted_list = {}
7
8
locations = {
9
	"IL": ["64.79.106.219", "64.79.106.220", "64.79.106.221"],
10
	"CA": ["nitrogen.bluevm.com", "oxygen.bluevm.com", "phosphorus.bluevm.com", "carbon.bluevm.com", "helium.bluevm.com", "lithium.bluevm.com", "neon.bluevm.com", "argon.bluevm.com"],
11
	"ATL": ["calcium.bluevm.com", "magnesium.bluevm.com", "silicon.bluevm.com", "sodium.bluevm.com"],
12
	"TX": ["nickel.bluevm.com"]
13
}
14
15
new_servers = {
16
	"IL": OrderedDict({12: 1, 34: 1, 56: 1}),
17
	"CA": OrderedDict({12: 2, 34: 3, 56: 1}),
18
	"ATL": OrderedDict({12: 1, 34: 2, 56: 1}),
19
	"TX": OrderedDict({12: 1, 34: 1, 56: 1})
20
}
21
22
new = {}
23
24
for location, x in new_servers.iteritems():
25
	sorted_list[location] = OrderedDict({})
26
	sorted_list[location][12] = []
27
	sorted_list[location][34] = []
28
	sorted_list[location][56] = []
29
30
for vps in reader:
31
	username, vpsid, current, plan = vps
32
	
33
	location = ""
34
	
35
	for loc, hosts in locations.iteritems():
36
		if current in hosts:
37
			location = loc
38
	
39
	if location == "":
40
		print "No location specified for %s" % (repr(vps))
41
		continue
42
	
43
	if "blue1" in plan or "blue2" in plan or "lebletspecial" in plan or "vps1" in plan or "vps2" in plan:
44
		sorted_list[location][12].append((username, vpsid))
45
	elif "blue3" in plan or "blue4" in plan or "vps3" in plan or "vps4" in plan:
46
		sorted_list[location][34].append((username, vpsid))
47
	elif "blue5" in plan or "blue6" in plan or "presale" in plan or "vps5" in plan or "vps6" in plan:
48
		sorted_list[location][56].append((username, vpsid))
49
50
for location, items in new_servers.iteritems():
51
	new[location] = OrderedDict({})
52
	
53
	for category, count in items.iteritems():
54
		for i in xrange(0, count):
55
			new[location][i] = []
56
57
for location, items in sorted_list.iteritems():
58
	current_server = 0
59
	
60
	for category, vpses in items.iteritems():
61
		total_vpses = len(vpses)
62
		total_servers = new_servers[location][category]
63
		per_server = total_vpses / total_servers
64
		
65
		for i in xrange(0, total_servers):
66
			start = per_server * i
67
			end = (per_server * (i + 1)) - 1
68
			
69
			if end + 2 >= total_vpses:
70
				end = total_vpses - 1
71
			
72
			current_server += 1	
73
			hostname = "s%d.c%d.%s.bluevm.com" % (current_server, category, location.lower())
74
				
75
			for s in xrange(start, end + 1):
76
				print "%s,%s,%s" % (vpses[s][0], vpses[s][1], hostname)

+ 41 - 0
script/status.py

@ -0,0 +1,41 @@
1
#!/usr/bin/python
2
3
import oursql
4
5
db = oursql.connect(host="localhost", user="root", passwd="", db="transfer")
6
c = db.cursor()
7
8
c.execute("SELECT * FROM servers")
9
servers = c.fetchall()
10
11
for server in servers:
12
	c.execute("SELECT * FROM entries WHERE `TargetNode` = ?", (server[1],))
13
	vpses = c.fetchall()
14
	
15
	total_vpses = len(vpses)
16
	
17
	if server[2] == 0 or server[2] == 3:
18
		current_vps = int(server[3])
19
	else:
20
		current_vps = int(server[3]) - 1
21
		
22
	if server[2] == 0:
23
		status = "Reported done."
24
	elif server[2] == 1:
25
		status = "Busy..."
26
	elif server[2] == 2:
27
		status = "Reported failure."
28
	elif server[2] == 3:
29
		status = "Finished."
30
	elif server[2] == 4:
31
		status = "Aborted."
32
	
33
	if server[2] == 1:
34
		c.execute("SELECT * FROM entries WHERE `TargetNode` = ? AND `Finished` = 1", (server[1],))
35
		current = c.fetchall()[0]
36
		current_vps_data = "\t%.2fG" % ((current[7] / 1024.0 / 1024),)
37
		status = "Busy... (%s)" % current[2]
38
	else:
39
		current_vps_data = ""
40
	
41
	print "%s%s%d/%d (%d%%)%s" % (server[1].ljust(25), status.ljust(25), current_vps, total_vpses, (100.0 * current_vps / total_vpses), current_vps_data)