Difference between revisions of "User:Tomato"

From ArchWiki
Jump to: navigation, search
m (Log parser: fix formatting)
 
(31 intermediate revisions by the same user not shown)
Line 1: Line 1:
just my personal scrapbook
+
''just my personal scrapbook''
  
=== Configure file sharing from KDE4 using samba ===
+
== lessfs ==
Sadly, this functionality is implemented using SUID perl script, and as Archlinux does not support SUID scripts nor it has perl-suid package we will have to get our hands dirty.
+
=== Sizes of different databases in lessfs ===
 +
With a 64k blocksize administration overhead would be approx 5~10% of the size of blockdata.tch
 +
=== Good all-round mount options ===
 +
<pre>-o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino\
 +
      ,default_permissions,allow_other,big_writes,max_read=65536,max_write=65536</pre>
  
{{Note|Works with simple file sharing only at the moment, advanced file sharing still asks for root password}}
+
=== Filesystem, mounting but not working ===
 +
Each and every partition with datafiles need 10% of free space, lessfs will freeze all writes to the partition for as long as the free space isn't aviable.
  
Work plan:
+
Version 1.0.4 has new config parameters:
# install packages
+
<pre>
# configure samba
+
MIN_SPACE_FREE=2
# modify {{Codeline|fileshareset}} scripts
+
MIN_SPACE_CLEAN=5
# create C wrapper for {{Codeline|fileshareset}} to work as a SUID binary
+
CLEAN_PROGRAM=/tmp/clean.sh
# configure KDE side of things
+
</pre>
# share a folder
+
<code>MIN_SPACE_FREE</code> defines minimum space left (in percent?) on the device, <code>MIN_SPACE_CLEAN</code> defines the amount of space left that triggers the <code>CLEAN_PROGRAM</code>
  
Packages we will need:
+
=== Bucket sizes ===
* {{Package Official|kdebase-dolphin}} this is the interface we will use to share folders
+
Usually when performance drops after reaching a certain database size this indicates that the bucketsize selected in <code>/etc/lessfs.cfg</code> is to low. The default 'lessfs.cfg' example included with the code only allows the database to grow up to: 2GB after which performance degrades.
* {{Package Official|samba}} for the server that will share files
+
To store 50GB the bucketsize should be at least : 27262976
* {{Package Official|kdenetwork-filesharing}} for System Settings File Sharing part
+
<pre>
* {{Package Official|kdebase-runtime}} basic KDE workspace (but if you're reading this, chances are you already have it installed)
+
2MB bucketsize equals 1000000 database entries.
* {{Package Official|kdelibs3}} for {{Codeline|fileshareset}} (not sure about it, but as of KDE4.4.0 it's still a dependency)
+
To store 50GB :
* {{Package Official|gcc}} to compile SUID wrapper
+
50GB = 50*1024*1024*1024 = 53687091200 bytes
 +
53687091200/4096 (4k blocksize) = 13107200 entries in the databases.
 +
13107200/1000000 (divide by 10^6 entries)
 +
13 *2 (10^6 entries requires 2MB) = 26
 +
26 * 1024*1024 = 27262976 -> This is the bucketsize you need for 50GB with a 4k blocksize.
 +
</pre>
 +
(from http://sourceforge.net/projects/lessfs/forums/forum/932600/topic/3304815)
  
==== Installing requisites ====
+
If you ever need to change the _BS settings, just unmount lessfs, change the settings, mount lessfs and run defrag.
Install all packages and their dependencies:
 
  
{{Codeline|pacman -S kdebase-dolphin samba kdenetwork-filesharing kdebase-runtime kdelibs3}}
+
=== Files to put on different spindles ===
 +
For max performance compile with –with-sha3 and put the FILEBLOCK and BLOCKUSAGE databases each on there own raid one disk set. The faster these disks are the better. BLOCKDATA can be on raid5/6.
 +
(from http://www.lessfs.com/wordpress/?p=295)
  
Log out and log in again (kdelibs3 are in non standard place and so the <code>PATH</code> must be updated)
+
=== Log parser ===
 +
<pre>
 +
grep 'lessfs\[8118\]' /var/log/messages.log | grep -Eo \
 +
  'size .* bytes \: lzo compressed bytes .* and .* duplicate blocks' |\
 +
  awk ' { SUM += $2 ; SUMCP += $8 ; DUP += $10 } \
 +
  END { print "total size " SUM/1024/1024/1024 "GiB, compressed size " \
 +
  SUMCP/1024/1024/1024 "GiB (ratio 1 : " SUM/SUMCP ") with " DUP\
 +
  " duplicate blocks (" DUP*64/1024/1024 "GiB deduplicated)" }' </pre>
  
==== Configuring samba ====
+
change <code>lessfs\[8118\]</code> to the PID of lessfs you want to monitor and <code>DUP*64</code> to the block size in kiB you use in it.
A basic config file works OK, though you'll need to create Samba users (with <code>smbpasswd -a &lt;user name&gt;</code> as root):
 
{{File|name=smb.conf|content=[global]
 
workgroup=HOME
 
server string = Samba Server
 
log file = /var/log/samba/&m.log
 
max log size = 50
 
load printers = No
 
dns proxy = No
 
  
[homes]
+
== Configuring Samba Domain/File server. ==
comment = Home Directories
 
read only = No
 
browsable = No
 
browseable = No
 
}}
 
  
Alternatively you can configure samba with <code>security = share</code> and add <code>guest account = &lt;your user name&gt;</code> to get an anonymous Samba server
+
=== What we want ===
 +
basically, everything that Samba has to offer:
 +
* save and share files (duh)
 +
* edit files/folders ACLs to the maximum extent possible for Samba<br />http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/AccessControls.html
 +
** mapping of permission inheritance
 +
* save file attributes (archive, system, hidden, read only) on samba server without screwing UNIX permissions
 +
* have the ability to manage users/computers in domain without the need to use UID 0 (root) account<br />http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/rights.html
 +
* automatic creation of user folders
 +
* Previous Version share feature
 +
* enforcement of log-on hours, password complexity, password age and requests for password change, and group policies <br />http://wiki.samba.org/index.php/Implementing_System_Policies_with_Samba<br />http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/PolicyMgmt.html
 +
* view and set file system quota
 +
* time server/syncronisation
 +
* printing with generic drivers to CUPS (optional)
  
Remember to add samba to <code>DAEMONS</code> in <code>/etc/rc.conf</code> if you want it to start at boot time.
+
guide:
 
+
https://help.ubuntu.com/community/SettingUpSambaPDC
==== Modifying fileshareset scripts ====
 
{{Command|name=|output=for directory in /opt/kde/bin/ /usr/lib/kde4/libexec; do
 
  sed 's/\/init.d\//\/rc.d\//g' fileshareset > fileshareset.pl
 
  rm filesharelist
 
  ln -s fileshareset.pl filesharelist
 
done}}
 
 
 
==== Create C wrapper ====
 
Create such files in your home directory:
 
 
 
{{File|name=kde4.c|content=#include <stdio.h>
 
#include <stdlib.h>
 
#include <string.h>
 
#include <sys/types.h>
 
#include <unistd.h>
 
 
 
#define EXEC "/usr/lib/kde4/libexec/fileshareset.pl"
 
 
 
int
 
main(int argc, char** argv)
 
{
 
  char **args;
 
 
 
  setegid(0);
 
  seteuid(0);
 
  setgid(0);
 
  setuid(0);
 
 
 
  args = calloc(argc+1, sizeof(char**));
 
  if (args == 0)
 
    exit(1);
 
 
 
  for (int i=0; i < argc; i++)
 
  {
 
    args[i] = calloc(1,strlen(argv[i]));
 
    if(args[i] == 0)
 
      exit(1);
 
    strcpy(args[i], argv[i]);
 
  }
 
  return execv(EXEC, args);
 
}
 
}}
 
 
 
{{File|name=kde3.c|content=#include <stdio.h>
 
#include <stdlib.h>
 
#include <string.h>
 
#include <sys/types.h>
 
#include <unistd.h>
 
 
 
#define EXEC "/opt/kde/bin/fileshareset.pl"
 
 
 
int
 
main(int argc, char** argv)
 
{
 
  char **args;
 
 
 
  setegid(0);
 
  seteuid(0);
 
  setgid(0);
 
  setuid(0);
 
 
 
  args = calloc(argc+1, sizeof(char**));
 
  if (args == 0)
 
    exit(1);
 
 
 
  for (int i=0; i < argc; i++)
 
  {
 
    args[i] = calloc(1,strlen(argv[i]));
 
    if(args[i] == 0)
 
      exit(1);
 
    strcpy(args[i], argv[i]);
 
  }
 
  return execv(EXEC, args);
 
}
 
}}
 
 
 
compile them using <code>gcc kde4.c -std=c99 -o kde4</code> and <code>gcc kde3.c -std=c99 -o kde3</code>.
 
Copy them to system folders:
 
{{Command|name=copying wrappers|output=cp kde3 /opt/kde/bin/fileshareset
 
chown root:root /opt/kde/bin/fileshareset
 
chmod u+s /opt/kde/bin/fileshareset
 
cp kde4 /usr/lib/kde4/libexec/fileshareset
 
chown root:root /usr/lib/kde4/libexec/fileshareset
 
chmod u+s /usr/lib/kde4/libexec/fileshareset
 
}}
 
 
 
==== Configure KDE ====
 
Right click any folder in your home directory and select Properties. Go to Share tab and click 'Configure File Sharing...', you'll be asked for root password.
 
(alternatively run <code>kcmshell4 fileshare</code> as root)
 
 
 
In the dialog that displays you need to select checkbox near "Enable Local Network File Sharing" and second one besides 'Simple sharing' (this one should be already selected).
 
 
 
Click 'Allowed Users' and select the second option -- one where users only from specified group can share files. Create new group, call it <code>samba-share</code>, add yourself, and any other user you might want to be able to share files.
 
 
 
Click OK until you close all the dialog boxes.
 
 
 
==== Share a folder ====
 
When the file sharing is configured you will see a new radio button on the Share tab -- Shared and Not shared. Change position to Shared, click OK. On the shared folder should appear now a small globe to indicate that it is shared.
 

Latest revision as of 11:40, 6 March 2010

just my personal scrapbook

lessfs

Sizes of different databases in lessfs

With a 64k blocksize administration overhead would be approx 5~10% of the size of blockdata.tch

Good all-round mount options

-o kernel_cache,negative_timeout=0,entry_timeout=0,attr_timeout=0,use_ino,readdir_ino\
      ,default_permissions,allow_other,big_writes,max_read=65536,max_write=65536

Filesystem, mounting but not working

Each and every partition with datafiles need 10% of free space, lessfs will freeze all writes to the partition for as long as the free space isn't aviable.

Version 1.0.4 has new config parameters:

MIN_SPACE_FREE=2
MIN_SPACE_CLEAN=5
CLEAN_PROGRAM=/tmp/clean.sh

MIN_SPACE_FREE defines minimum space left (in percent?) on the device, MIN_SPACE_CLEAN defines the amount of space left that triggers the CLEAN_PROGRAM

Bucket sizes

Usually when performance drops after reaching a certain database size this indicates that the bucketsize selected in /etc/lessfs.cfg is to low. The default 'lessfs.cfg' example included with the code only allows the database to grow up to: 2GB after which performance degrades. To store 50GB the bucketsize should be at least : 27262976

2MB bucketsize equals 1000000 database entries.
To store 50GB :
50GB = 50*1024*1024*1024 = 53687091200 bytes
53687091200/4096 (4k blocksize) = 13107200 entries in the databases.
13107200/1000000 (divide by 10^6 entries)
13 *2 (10^6 entries requires 2MB) = 26
26 * 1024*1024 = 27262976 -> This is the bucketsize you need for 50GB with a 4k blocksize.

(from http://sourceforge.net/projects/lessfs/forums/forum/932600/topic/3304815)

If you ever need to change the _BS settings, just unmount lessfs, change the settings, mount lessfs and run defrag.

Files to put on different spindles

For max performance compile with –with-sha3 and put the FILEBLOCK and BLOCKUSAGE databases each on there own raid one disk set. The faster these disks are the better. BLOCKDATA can be on raid5/6. (from http://www.lessfs.com/wordpress/?p=295)

Log parser

grep 'lessfs\[8118\]' /var/log/messages.log | grep -Eo \
  'size .* bytes \: lzo compressed bytes .* and .* duplicate blocks' |\
  awk ' { SUM += $2 ; SUMCP += $8 ; DUP += $10 } \
  END { print "total size " SUM/1024/1024/1024 "GiB, compressed size " \
  SUMCP/1024/1024/1024 "GiB (ratio 1 : " SUM/SUMCP ") with " DUP\
  " duplicate blocks (" DUP*64/1024/1024 "GiB deduplicated)" }' 

change lessfs\[8118\] to the PID of lessfs you want to monitor and DUP*64 to the block size in kiB you use in it.

Configuring Samba Domain/File server.

What we want

basically, everything that Samba has to offer:

guide: https://help.ubuntu.com/community/SettingUpSambaPDC